Upgrade did actually complete. pictrs had some issues but I think I fixed them (hopefully).

Any issues let me know.

Site will be down for the upgrade sometime today as I run the scripts we have and see what happens and then troubleshoot. Hopefully not too long.

Lemmy once again needs upgrading. This time all components - pictrs, postgres and lemmy itself. This will mean that at least 1 of these (fragile) things will almost certainly break and leave the server down for some time. I will be investigating how best to unbreak the server over the next couple of weeks and post when we intend to do the update that inevitably will destroy the server in some way. If you use the server daily you may wish to create a secondary account so that you are not too disrupted when the update happens. Unfortunately upgrading lemmy is not as easy as one might hope so I cant tell you exactly what will happen ahead of time. I will post again when we have a general idea what to do with an intended date.

Hi there,

Unfortunately the site was down for about 2 days over the 26-27 May this month.

This was due to the datacentre we use being affected by a storm/tornado: (https://apnews.com/article/tornadoes-weather-texas-oklahoma-93001e1d81f120d0d55c8dfa24938250) which took out power to the facility for just under 2 days. At no time was there any risk of loss to the server due to off site backups, however we did have to wait for the centre to come back online.

While this server is run extremely cheaply, at just $20 USD per month (approx) which includes paying for the compute, domain, email, and off site backup as well as object storage, it does mean that there is not redundancy when a site wide outage occurs. The low cost means that there is no risk of this server ever being unable to meet its costs and be paid for, but it does mean we sometimes can have an outage if a major event happens such as what happened and a delay before being brought back online.

The decision for this was because this server is designed to help reduce load on the rest of the fediverse and does not store much local content so a couple days a year the server is out was deemed acceptable during the design phase.

If you have any questions or comments feel free to post and I will try my best to answer them :)

• P

Upgrade to 0.19.3 was completed with no issues today (from 0.19.1)

Just want to give everyone an apology for the downtime last few days. Its unfortunate that we were down around 18 - 20 hours in total and rest assured I have made improvements based in what happened to reduce this going forward. Its been 24h now since the upgrade to 0.19.1 and everything has remained stable so far.

Hopefully thats the end of it. Merry Christmas and a Happy New Year to you all :)

-P

Another apology to everyone.

I upgraded the instance to 0.19.1 today, and the upgrade unfortunately had a major issue. This caused the Database to become fairly broken and it has taken several hours to repair the damage and the database.

I have found the root cause of the issue and will be implementing some changes to the backup routine to avoid this issue in the future for future upgrades. This is thanks to some very welcome help from the admins on the Beehaw instance who are incredible and supportive.

As a result this will be improved and shouldnt happen again in the future when upgrading. I still need to run a vacuum on the postgres database but I will do this tomorrow, but there will be a 15 minute window tomorrow when the site is unresponsive while this is run.

Hopefully after that there are no more issues and the site can finally come back to being reliable again.

Kind regards,

• P

Unfortunately the server went offline today for approximately 12 hours. This is due to a deficiency in the backup routine, where if the vps hoster I use turns off the server/has an interruption, during the window of about 30 minutes where the backup runs then the script does not put everything back in place to ensure the server can start up again.

The good news is, this new issue that was not thought about should be resolvable with a reboot script, that uses an if or statement to check if the backup did do what it was supposed to and if not take some fairly simple logic to put things back as they should be.

In the next couple of weeks I will implement and test such a solution.

However it does raise another issue in that lemmy had some growth and then shrinking, that has stabilised and left us with a constant but slow increase in users, lost some of the admins that used to help check this server and they have abandoned their lemmy deployments for whatever reason and there are less people around actually checking of things are working.

Sadly I didnt notice the server was offline for 12 hours as I was working and in meetings all day and only got around to checking lemmy later in the afternoon. This is disappointing as the fix only took a few minutes - less than 10 to login, check what happened and correct it.

Had I been made aware of an issue earlier, many hours of downtime would have been saved.

To be clear, this server is run extremely cheaply. This means we dont have as many automated checks as other more expensive lemmy servers and we must rely on our members to tell us if things go wrong. This design choice was chosen in case lemmy becomes less popular and so the cost of the server is no issue. Essentially I can pay and run it out of my pocket just as a public service.

What would be great is if anyone can join the admin team to do some actual man hours work like, respond to new users joining the server, checking its functional and messaging me if not, blocking a user here or there if we get a complaint and so on. All of these tasks have become harder as the admins we used to rely on together have lost interest.

Finally if nobody is able to help, and the expectation is this simply continues as is for free indefinitely then that is totally fine also. The server is in no danger of being turned off or anything like that, requests and issues that come up will just take longer to be fixed as a result. To be clear I dont mind, if I cant use the lemmy server I just login to it and fix it and then carry on my merry way. This mostly affects people who cant do that.

Just wanted to make an appeal for some admins to improve the areas we are weakest but also clarify nobody is under and obligation to do anything and the server remains free regardless. The weakest area is not enough humans helping by messaging and opening communication lines if there are issues. If you can type a message and send it to someone, use this server and want to make it better then you are qualified to help.

Any help is of course greatly appreciated. My commitment is that you will never be expected to do anything to ‘keep it going’ but I also sleep, work, and so on so this is why I am asking. Totally up to whatever anyone feels comfortable with. Zero obligations.

Thanks -P