cross-posted from: https://lazysoci.al/post/39049419

lazysoci.al has been updated to 0.19.14

Let me know if any issues.

cross-posted from: https://lazysoci.al/post/38720861

As announced earlier this week, some maintenance will be performed causing several hours of downtime on Sunday 2025-12-07.

The following changes will be made:

• All front-ends will be migrated to a separate VM.
• pict-rs S3 storage will be migrated from minio to rustfs (minio is no longer being updated).
• Anubis will be implemented and placed in front of all front-ends.
• Cloudflare tunnel will be decommissioned.

The site will be offline for an extended period of at least 2 hours, perhaps longer. Backups are in-place so should there be major issues following these changes, they will be rolled back.

I'll pin this post on Sunday morning a couple of hours before the work starts. If real-life gets in the way, it'll be postponed to the following weekend.

cross-posted from: https://lazysoci.al/post/35188768

There shall be a 30 minute downtime after this post is 30 minutes old. ~ 12:40 UK time

The Proxmox cluster Lemmy is running on is being updated to version 9, and I'm doing all nodes at pretty much the same time, so all VMs are going off for the duration.

cross-posted from: https://lazysoci.al/post/34975344

Imgur have made the decision to block the UK following being threatened by the UK IPO due to the Online Safety Act.

Unfortunately this Lemmy server resides in the UK and is not unable to access imgur as a result. The image failures you may be expieriencing is due to this instance fetching images on your behalf to protect your internet identity (image_mode: ProxyAllImages)

I am working on a work-around which is almost ready to deploy. I expect images to stop functioning completely for a short while whilst I migrate pict-rs to another system (thank god I used S3 for pict-rs storage, else this migration would take days).

Update: pict-rs wasn't migrated, instead its outbound traffic has been routed out of another region. Unfortunately attempting to purge the cache of imgur images so it'll redownload is too much of a headache. The cache is set to 7-days but it seems setting it to 1-hour doesn't affect existing images, only new ones.

If you are in the UK, then new posts that use imgur.com images shall be viewable. However this will depend on your client. For example Boost for Android gets the image URL direct from the original server, so I see the the thumbnail OK (cached via lazysoci.al) and then the "imgur has blocked you" full image (due to Boost going direct).

I expect this will only get worse.

cross-posted from: https://lazysoci.al/post/25567708

I shall shortly be updating Lemmy to 0.19.11 which fixes a security issue. Downtime is expected to be minimal.

At the same time I shall also be updating all alternative frontends to their latest versions.

cross-posted from: https://lazysoci.al/post/24004627

Along with the upgrade to 0.19.10 today I have made the following additional changes

• Added Photon frontend
  • Based on the project at https://github.com/Xyphyn/photon
  • Available at
    • https://photon.lazysoci.al/
    • https://p.lazysoci.al/
• Added Voyager frontend
  • Based on the project at https://github.com/aeharding/voyager
  • Available at
    • https://voyager.lazysoci.al/
    • https://v.lazysoci.al/
• Added Alexandrite frontend
  • Based on the project at https://github.com/sheodox/alexandrite
  • Available at
    • https://alexandrite.lazysoci.al/
    • https://a.lazysoci.al/
• Added Tesseract frontend
  • Based on the project at https://github.com/asimons04/tesseract
  • Available at
    • https://tesseract.lazysoci.al/
    • https://t.lazysoci.al/
• Removed the Lemmy UI gearbox debug icon
  • Thanks for pointing this out @Blaze@lemmy.dbzer0.com @Blaze
• Created this community
  • Another blaze idea, keep them coming :)

When this post is 1 hour old, I shall be updating lazysoci.al to Lemmy verison 0.9.10

Downtime is expected to be ~10 minutes or less

This evening I have updated Lemmy from 0.19.8 to 0.19.9

There has been an increasing number of new users. Many of these have turned out to be dishonest trolls and subsequently banned. One of the traits of the internet I guess.

For those that remain, warm welcome :)

I shall be moving the pict-rs data from local filesystem to local S3 this evening.

The site shall be offline for an extended period during this migration. There is ~450G in just over 3 million files currently hosted on the pict-rs instance.

Lemmy and all its containers are on an NVMe backed zfs pool. With pict-rs fast approaching 500G I've been exploring alternatives. So I introduced an S3 service using spinning rust.

I expect the migration to take several hours so will start this just before I head off for bed between 2300-0000 GMT.

Update 2025-01-21 03:09 GMT

Downtime 3h 17m -- Migration completed. Images seem to be working, both new and old. Site is catching up on missed posts.

The new server is prepped and ready to take to the datacentre. I've created a vxlan (mikrotik eoip) to the office so have been transferring services to the new server when I had time over the last few days.

I just performed the transfer of Lemmy to the new server which was an unannounced 2h30m downtime (sorry!). It was a good backup restore test though.

To complete the project, I shall be visiting the datacentre on Thursday, 2025-01-02 to swap the old server for the new. lazysoci.al will be offline on 2025-01-02 from 0830 to 1200 GMT

Apologies for the inconvenience.

Update: server swap completed without issue.

When this post is 2 hours old (2100 UTC) I shall be performing maintenance on the pict-rs container and updating lazysoci.al to Lemmy 0.19.8.

During the maintenance period, the site shall be offline.

The Lemmy update should be straight forward, however I also need to purge old cached images in pict-rs which could take a few hours.

Edit: Site back online

Holidays are coming, and with that, spare time!

I shall be updating the instance over the next couple of weeks to 0.19.8. I'll post an update some hours before the upgrade.

Over the holidays I shall be decommissioning the physical server that this Lemmy instance is hosted on. The new server is being built and will be a Dell PowerEdge R640 running all U.2 NVMe drives. Hopefully, this will make the server the fastest it can possibly be.

Our lemmy instance stopped processing new activity sometime on Monday 2024-11-18 morning.

The root cause remains unknown. Services were online. Database was responsive.

Lemmy server logs were showing the incoming ActivityPub requests, no errors, but no response was being returned to the sender. The system was restarted on 2024-11-19 and processing of requests resumed.

Luckily, the protocol allows for some caching of requests across all servers, so after 30 minutes of heavy load, our server had mostly caught up.

I was away on Monday, and I did notice the issue, but I initially thought it was a problem with my mobile app (recently moved to Boost). I normally view Lemmy sorted by "Top - Last Twelve Hours" and on Tuesday this returned zero results, which prompted a closer look.

I have added additional monitoring to the system, checking for the age of the latest post. I shall now receive an alert if a new post has not been received for 15 minutes. This may result in some false positives if Lemmy is quiet so may adjust this in the future.

Example data:

A status page is available at https://status.lazysoci.al/

Our hosting provider had an outage which lasted ~30 minutes.

I raised a support case as soon as I got the alert as I couldn't access the systems via their link, but the 4G backup confirmed servers were online.

lazysoci.al was offline whilst the cluster was being updated.

Outage was 38 minutes. 11:00 to 11:38 BST

This was expected to be <15 minutes. The extended outage was due to an issue bringing up a docker container which was a pre-requisite for the load balancer.

I shall be migrating pict-rs to postgres, now.

The pict-rs service shall need to be offline which shall effect thumbnail generation and the ability to post images.

Should be completed within 4 hours of this post.

Edit: Migration has completed at 1515 UK time.

Registration requirements

I've had to ban multiple vile accounts this morning. It seems this instance has found itself on the radar of trolls.

To that end, sign-ups now require users to fill in a questionnaire prior to joining. I always thought it was lame, and it won't really prevent a troll account from joining, but it would slow them down, and likely cause them to find another easier-to-join instance.

Upgrade

I shall be upgrading the instance to ~~0.19.4~~ 0.19.5 ~~this afternoon~~ on Friday morning, so expect a little downtime

I'll unpin this notice once the update is complete.

Edit: Unfortunately work got in the way of me performing the update today, so postponing this to first thing tomorrow morning.

Cached images

We use the standard group of services for Lemmy, including the pict-rs image/thumbnail cache. This image cache grew to 700G recently and continues to grow as Lemmy grows. Therefore some effort has been made to keep it under control.

Cached images from other instances are now only kept for 90-days. This doesn't remove the original image.

Upgraded to 0.19.2

Upgrade itself went OK. Also updated pict-rs to 0.5.1 which performed a metadata update.

I tested this upgrade last Thursday 2024-01-11 but rolled back when outgoing federation didn't work.

Found a fix from matrix.org (thanks to Ategon@programming.dev and Illecors@lemmy.cafe) by updating the "updated" column on the instance table.

UPDATE instance SET updated = now() WHERE updated > now() - interval '14 days';

Total downtime was 12 minutes.

Upgraded to 0.19.0

Looks good so far. Took a snapshot prior to upgrade, but initial monitoring shows no issues.

Total downtime was 42 minutes.

What was meant to be a quick blip ended up being over an hour. Migrating the reverse proxy that sits in front of the Lemmy server failed as dockerhub is having an outage.

Overview

lazysoci.al was offline for 3h 15m today following a database corruption. Server is now back online, federated data is flowing again.

Details

I moved the server to its own dedicated host this morning, for both the performance and security (dedicated vlan) impact. Should have been a simple case of moving the virtual disk with the Lemmy data to the new VM and spinning up the new docker image.

The docker logs didn't show any initial issues, however writing to the database gave errors of ERROR: relation "approvals" does not exist for every UPDATE query.

After some troubleshooting, I finally thought the database was corrupted, so I started a restore from last nights backup. This took approx. 2h 30m to restore.

Post-restore, the same issue. I then performed an update to the latest beta, and the issue is now resolved.

This has highlighted one problem. I use proxmox-backup-server and proxmox-virtual-environment. You can't easily restore a single disk from a VM into a ZFS volume. If using the web interface you have to restore the entire VM. So the backup took much longer to restore, as it needed to restore several other disks first.

Improvements

• The new setup would restore quicker as there is only the OS and data, unique to Lemmy.
• There shall now be backups performed every 2 hours instead of nightly.
• A new script performs a snapshot every hour, and retains snapshots for 24 hours. Snapshots can be restored pretty much instantly.

I'll also do some testing with the CLI proxmox-backup-client so I can work out how to restore a single disk into a zvol.