endlesstalk.org

Upgrading to 0.19.11, which fixes a security vulnerability and a couple of new features. Release notes here for more information.

For this upgrade I expect there to be no or very minimal downtime(5min or less). However, if any major issues occurs during the upgrade, you can check the uptime site here or site status here

The local time for the update can be seen here

EDIT: I had to do the update a bit later, than expected, but it is now updated. There was an issue, causing some downtime. The cause was that the database slots filled up, but it is now fixed.

The upgrade went smoothly, but was a bit delayed.

Let me know, if there are any issues after the upgrade!

Upgrading to 0.19.9, which contains a minor performance improvement and some bugfixes. Release notes here for more information.

For this upgrade I expect there to be no or very minimal downtime(5min or less). However, if any major issues occurs during the upgrade, you can check the uptime site here or site status here

The local time for the update can be seen here

This will cause downtime, but I don't know for how long. An estimate is around 1-2 hours or so, but it is hard to say.

I apologize for the short warning.

Local time

This will cause downtime, but I don't know for how long. An estimate is around 1-2 hours or so, but it is hard to say.

I apologize for the short warning.

Local time

EDIT: While pictrs are migrating, images ~~might not work(Works for me currently though)~~ will not work EDIT2: Done

The upgrade went smoothly.

Let me know, if there are any issues after the upgrade!

The upgrade went smoothly.

Let me know, if there are any issues after the upgrade!

Finally upgrading to 0.19.8(I have been a bit busy), which contains a few minor fixes and improvements. Release notes here for more information about the bugfixes.

For this upgrade I expect there to be no or very minimal downtime(5min or less). However, if any major issues occur during the upgrade, you can check the uptime site here or site status here

Local time can be seen here

EDIT: Moved date for the upgrade to 26th January, same time.

There was an issue with the database, since the migration somehow took 200GB extra space, which caused the database on one of the server to go down. The database switched over to its replica, so it works now, but there is a period of 30 mins, where some data will have been lost.

Let me know, if there are any other issues.

EDIT: There does seem to be an issue with creating a post. I will look into it.

EDIT2: Fixed.

Upgrading to 0.19.7, since it contains some fixes discovered after 0.19.6 have been released . Release notes here for more information about the bugfixes.

For this upgrade there will be downtime and I expect it to last around 30 min but it all depends on how long the database migrations will take. If there are any major issues with the upgrade, you can check the uptime site here or site status here

Local time can be seen here

0.19.6 brings a lot of changes. See release notes here for more information.

For this upgrade there will be downtime and I expect it to last around 30 min but it all depends on how long the database migrations will take. If there are any major issues with the upgrade, you can check the uptime site here or site status here

Local time can be seen here

Images aren't loading and I'm not sure what the cause is, after looking into it shortly.

Will look more into, when I have the time.

EDIT: Pictrs db was missing a lots of data. I'm unsure how the data was lost. I have now restored from a backup from 4 days ago(Earlier backups didn't contain data), so images for the last 4 days will be lost.

This should however be the last time for a long time, since I have greatly improved the setup.

Pictrs database got corrupted during the process, which is why the images for the last 6 days are lost.

Let me know, if there are any issues after moving to the new host.

The database was corrupted, so had to recover from backup. About 6 hours of data was lost.

~~There does seem to be a weird network error sometimes. I'm looking into it.~~ FIXED

Lastly I apologize for all the downtime lately.

So I had to migrate to a new host again. The response times are a bit slow and spikes sometimes. I'm looking into it.

EDIT: Should be a bit better now, but there are still some spikes in response times, so I will keep looking into it.

Endlesstalk has been migrated to a new host.

There did seem to have been some caching issues, causing some certificate issues, but it seems to be fixed(Atleast for me).

Let me know, if there are any issues after the migration

EDIT: The 525 ssl certificate error shows up intermittently. I'm looking into it.

EDIT2: Fixed. Was an issue with the nginx loadbalancer.

Earlier today one of the servers, where endlesstalk is hosted went down. After some time, the server came back up again, but there were some unknown issue and the server was unstable. So preparation to migrating endlesstalk to a new host began. However after setting the new servers up, there was success with getting one of the "old" servers up and running again.

Tommorow at ~~18:00~~ 20:00 UTC the migration to the new host will begin. See local time here. There will be some downtime with this, probably around an hour or less.

EDIT: Server went down again, but should be back again now.

EDIT2: 20:00 UTC, since I forgot I have something from 17-19 UTC.

The upgrade went smoothly and everything seems to work.

Let me know, if there is anything that dosn't work after the upgrade.

I have found the issue with the database migration, so the upgrade to the latest version of lemmy can proceed.

0.19.5 brings a lot of smaller bugfixes. See release notes here for more information. I will also upgrade the database to a newer version(postgres 16).

For this upgrade there will be downtime and I expect it to last around 1 hour or less. If there are any major issues with the upgrade, you can check the uptime site here or site status here

Local time can be seen here

The database migration to 0.19.4 failed, because the database schema doesn't align with the state the migrations want. The reason is probably because it didn't restore correctly from a previous backup, but I don't actually know the cause.

I thought I could create a new database with a correct schema and then import the data from the currrent database into the new one. This might still be possible, but it simply takes too long and it has gotten too late for me(03:00 in the night).

I will look into a fix for the migration and when I have a fix I will announce a new date for the upgrade to 0.19.4.

0.19.4 brings a lot of changes. See release notes here for more information.

There should be no downtime or very minimal downtime. If there are any issues, check the uptime site here or site status here

Local time can be seen here

Note: An update to postgres 16 and pictrs 0.5 is also comming soon, which will bring some downtime. Don't know when yet, but will post an update, when I know.

EDIT: There was an issue with migrating the database, while upgrading to 0.19.4, so will take longer.

EDIT2: The database is in a different state, than the migration to 0.19.4 expects. The cause is not clear, but I'm looking into it.

Hello

I have noticed that the server have been going down a lot for 10 - 20 mins.

Unfortunately, I'm currently on vacation, so I don't think I will have the time to fix it.

I will be back tomorrow evening and will look into it and hopefully fix it then.

EDIT: There was a misconfiguration of the auto scaling setup. This scaled the system up and used all of its CPU, which caused the site to be unresponsive.

This should be fixed now, but I will keep monitoring it.

While working on a small fix to lemmy, that was causing some unneeded cpu usage, I made a change that unfortunately caused the db and pictrs service storage to be deleted.

Thankfully I have backups of everything, so I went to restoring from a backup. However the restoring was very slow, since I used an unoptimal way to backup the db(raw sql dump). After the first backup completed, I found out that it was missing data, so I tried an older backup, but that didn't work either. It was missing data as well. So I tried a backup from another server(Since I backup to 2 different servers), which finally worked.

Usually restoring from backups, haven't taken too long previously, since my backups are faily small, but I will need to look into a quicker way to restore backups for lemmy, since the backup size of lemmy is much bigger.

NOTE: Data from ca. 2 hours before the site went down(16-18 UTC) will be missing and I'm unable to restore it.

The s3 host that pictrs use, have gone down.

I might have to move to another s3 host, which will mean it will take a bit before images are working. This will cause a loss of images for the last 2-3 hours before pictrs stopped working.

EDIT: Have moved to new s3 host. Unsure how many images were lost during the outage.

Local time can be seen here

0.19.3 is mostly bugfixes. See release notes here for more information.

There should be no downtime or very minimal downtime. If there are any issues, check the uptime site here or site status here