lazyadmin

I gave the new VM for all the frontends only 2G ram, doh.

Backend continued functioning.

cross-posted from: https://lazysoci.al/post/39049419

lazysoci.al has been updated to 0.19.14

Let me know if any issues.

lazysoci.al has been updated to 0.19.14

Let me know if any issues.

cross-posted from: https://lazysoci.al/post/38720861

As announced earlier this week, some maintenance will be performed causing several hours of downtime on Sunday 2025-12-07.

The following changes will be made:

• All front-ends will be migrated to a separate VM.
• pict-rs S3 storage will be migrated from minio to rustfs (minio is no longer being updated).
• Anubis will be implemented and placed in front of all front-ends.
• Cloudflare tunnel will be decommissioned.

The site will be offline for an extended period of at least 2 hours, perhaps longer. Backups are in-place so should there be major issues following these changes, they will be rolled back.

I'll pin this post on Sunday morning a couple of hours before the work starts. If real-life gets in the way, it'll be postponed to the following weekend.

As announced earlier this week, some maintenance will be performed causing several hours of downtime on Sunday 2025-12-07.

Edit: This maintenance shall begin at 1730 UK time (2 hours from this edit)

The following changes will be made:

• All front-ends will be migrated to a separate VM.
• pict-rs S3 storage will be migrated from minio to rustfs (minio is no longer being updated).
• Anubis will be implemented and placed in front of all front-ends.
• Cloudflare tunnel will be decommissioned.

The site will be offline for an extended period of at least 2 hours, perhaps longer. Backups are in-place so should there be major issues following these changes, they will be rolled back.

I'll pin this post on Sunday morning a couple of hours before the work starts. If real-life gets in the way, it'll be postponed to the following weekend.

Update 2025-12-07: Maintenance has ended, however somewhat failed.

I want to wait until this rustfs issue is resolved prior to migrating. I tried to perform some pruning of the data post migration, but noticed directories not being removed. I didn't see this until I had already started the sync. At least now I know a re-sync between the two S3 buckets takes about 3 hours.

Implementing Anubis is going to require some additional work to create a good policy configuration file and I ran out of time this evening.

The lemmy UI's are now on a new VM, and Cloudflare has been removed, so there is that, but more work needed.

A short update as things have been quiet lately.

Image proxy

A few weeks ago I made what was meant to be a temporary configuration change to fix imgur.com images. I set pict-rs proxy cache lifetime to 1h in an attempt to quickly have the system re-download failed images. I forgot to remove this :(

The implications were that images were slow to load on the site as pict-rs had to make its own fetch. This has now been resolved and at least for me the site is running smoother for it.

Bots and Cloudflare

You may have noticed that the Cloudflare outage the other week affected the site. I have been using Cloudflare as an attempt to reduce bot-spam, however it isn't very effective.

Whilst the site is on good hardware, I do get regular performance alerts due to scrapers hitting the site. At times there are ~5000 concurrent connections. It usually only lasts a few minutes and whilst the site does slow down a little, it isn't unbearable.

In any case, I plan on exploring the use of Anubis https://anubis.techaro.lol/ as an alternative to Cloudflare. This will (1) reduce reliance on large corporations (2) increase privacy as Cloudflare is a privacy risk (3) combat bot-scraping. It may also effect mobile clients, so I'll be testing all clients as part of the roll-out.

I'm expecting to work on this at the weekend, however I'll post another update beforehand.

That's all for now.

Apologies after the update this morning the docker host running the lemmy frontends (seperate from lemmy itself) was having real issues and took a while to resolve.

All back up and running now :)

Weird issue though - there was an unused container running squid (the forward proxy service) that was causing the system to freeze. The proxmox host node was getting a load average of 600+. Took a while to find it as had to start each container one, wait 2 minutes for it to settle, then try the next.

cross-posted from: https://lazysoci.al/post/35188768

There shall be a 30 minute downtime after this post is 30 minutes old. ~ 12:40 UK time

The Proxmox cluster Lemmy is running on is being updated to version 9, and I'm doing all nodes at pretty much the same time, so all VMs are going off for the duration.

There shall be a 30 minute downtime after this post is 30 minutes old. ~ 12:40 UK time

The Proxmox cluster Lemmy is running on is being updated to version 9, and I'm doing all nodes at pretty much the same time, so all VMs are going off for the duration.

Update: All back up and working

cross-posted from: https://lazysoci.al/post/34975344

Imgur have made the decision to block the UK following being threatened by the UK IPO due to the Online Safety Act.

Unfortunately this Lemmy server resides in the UK and is not unable to access imgur as a result. The image failures you may be expieriencing is due to this instance fetching images on your behalf to protect your internet identity (image_mode: ProxyAllImages)

I am working on a work-around which is almost ready to deploy. I expect images to stop functioning completely for a short while whilst I migrate pict-rs to another system (thank god I used S3 for pict-rs storage, else this migration would take days).

Update: pict-rs wasn't migrated, instead its outbound traffic has been routed out of another region. Unfortunately attempting to purge the cache of imgur images so it'll redownload is too much of a headache. The cache is set to 7-days but it seems setting it to 1-hour doesn't affect existing images, only new ones.

If you are in the UK, then new posts that use imgur.com images shall be viewable. However this will depend on your client. For example Boost for Android gets the image URL direct from the original server, so I see the the thumbnail OK (cached via lazysoci.al) and then the "imgur has blocked you" full image (due to Boost going direct).

I expect this will only get worse.

Imgur have made the decision to block the UK following being threatened by the UK IPO due to the Online Safety Act.

Unfortunately this Lemmy server resides in the UK and is not unable to access imgur as a result. The image failures you may be expieriencing is due to this instance fetching images on your behalf to protect your internet identity (image_mode: ProxyAllImages)

I am working on a work-around which is almost ready to deploy. I expect images to stop functioning completely for a short while whilst I migrate pict-rs to another system (thank god I used S3 for pict-rs storage, else this migration would take days).

Update: pict-rs wasn't migrated, instead its outbound traffic has been routed out of another region. Unfortunately attempting to purge the cache of imgur images so it'll redownload is too much of a headache. The cache is set to 7-days but it seems setting it to 1-hour doesn't affect existing images, only new ones.

If you are in the UK, then new posts that use imgur.com images shall be viewable. However this will depend on your client. For example Boost for Android gets the image URL direct from the original server, so I see the the thumbnail OK (cached via lazysoci.al) and then the "imgur has blocked you" full image (due to Boost going direct).

I expect this will only get worse.

When this post is ~90 minutes old, lazysoci.al will have a downtime of ~1 hour.

I am upgrading the cluster that hosts lemmy, upgrading both RAM and Proxmox on all nodes.

Update: 1230 GMT+1; All back up and running.