April is here!

So much has happened since the last update, we've migrated to a new server, we've failed to update to a new lemmy version, automated our rollouts, fought with OVH about contracts. It's been a lot.

Strap in for story time about the upgrade, or skip till you see the break for the next section.

So good news is that we are successfully on v0.19.11.

The bad news is that we had an extended downtime.

Recently I had some extra time to completely automate the rollout process so Reddthat didn't rely solely on me being on 1 specific computer which had all the variables that was needed for a deployment.
As some people know I co-manage the lemmy-ansible repository. So it wasn't that hard to end up automating the automation. Now when a new Version is announced, I update a file/files, it performs some checks to make sure everything is okay, and i approve and roll it out. Normally we are back online within 30 seconds as the lemmy "backend" containers do checks on start to make sure everything is fine and we are good to go. Unfortunately it never came back up.

So I reverted the change thinking something was wrong with the containers and the rollout proceeded to happen again. Still not up :'( Not having my morning coffee and being a little groggy after just waking up.

Digging into it our database was in a deadlock. Two connections were attempting to do the same but different which resulted in it being locked up and not processing any queries.

Just like Lemmy World, when you are "scaling" sometimes bad things can happen. re: https://reddthat.com/post/37908617.

We had the same problem. When rolling out the update two containers ended up starting at the same time and both tried to do the migrations instead of realising one was already doing them.

After quickly tearing it all down. We started the process of only having 1 container to perform the migration and then once that had finished starting everything else we were back online.

Going forward we'll probably have to have a brief downtime for every version to ensure we don't get stuck like this again. But we are back-up and everything's working.

Now for the scheduled programming.

OVH

OVH scammed me out of the Tax on our server renewal last month. When our previously 12 month contract was coming to the end we re-evaluated our finances and were found wanting. So we ended up scaling down to a more cost-effective server and ended up being able to pay in AUD instead of USD which will allow us to stay at a single known price and not fluctuate month to month.
Unfortunately I couldn't cancel the contract. The OVH system would not let me click terminate. No matter what I did, what buttons I pressed, or how many times I spun my chair around it wouldn't let me cancel. I didn't want to get billed for another month when we were already paying for the new server. So a week before the contract ended I sent a support ticket to OVH. You can guess how that went. The first 2 responses I got from them after 4 days was "use the terminate feature". They didn't even LOOK at the screenshots clearly outlining the steps I had taken and the generic error... So I get billed for another month... and then have to threaten them with legal proceedings. They then reversed the charge. Except for the Tax. So I had to pay 10% of the fee to cancel our service. Really unhappy with OVH after this ordeal.

Automated rollouts

I spent some time after our migration ensuring that we have another system setup which will be able to rollout updates. So we are not dependant on just me and my one random computer :P All was going very well until an upgrade with database migrations happened. We'll be working on that soon to make sure we don't have unforeseen downtime.

Final Forms

Now that the dust has settled and we've performed the migrations starting next month I'll probably go back to our quarterly updates unless something insane happens. (IE: Lemmy drops v1 👀 )

We also modified our "Reddthat Community and Support" community to be a Local Only community. The original idea for the community was to have a place where only reddthat could chat, but back when we started out that wasn't a thing! So now if you want to voice your opinion to other Reddthat users please feel free too knowing other instances won't come in and derail the conversation.

As a reminder we have many ways to donate if you are able and feel like it! A recurring donation of $5 is worth more to me than a once of $60 donation. Only because it allows me to forecast the year and work out when we need to do donation drives or relax knowing everything in it's current state will be fine.

• Open Collective (preferred)
• Kofi
• Litecoin Direct
• Monero Direct (preferred)

Cheers,

Tiff

We just successfully upgraded to the latest Lemmy version, 0.9.10, probably the last before the v1 release.

This addresses some of the PM spam that everyone has been getting. Now when that user is banned and we remove content it also removes the PMS. So hopefully you won't see them anymore!

Over the next couple days will be planning for our migration to our new server as our current server's contract has ended. I expect the down time to last for about an hour, if not shorter. You'll be able to follow updates for the migration by our status page at https://status.reddthat.com/

Normally this update would be a week in advance and more nicely formatted that turns out the contract ends on the 25th and I don't want to get charged for another month at a higher rate when I just purchased the new server.

See you on the other side,

Tiff

EDIT:

22 Mar 2025 02:42: I'm going to start the migration in 5 mins (@ 3:00)

22 Mar 2025 03:01: that was the fastest migration I've ever done. pre-seeding the server and Infra as Code is amazing!

We've turned off our crypt donation p2pool (as no-one was using it), and two of our frontends, alexandrite and next (for the same reasons)

Time to celebrate with some highly accurate Australian content:

Hello Reddthat! We are back for another update on how we are tracking. It's been a while eh? Probably because it was such smooth sailing!

In the middle of February we updated Lemmy to v0.19.9 which contained some fixes for federating between Mastodon and Lemmy so hopefully we will see less spam and more interaction from the larger mastodon community. While that in of itself is a nice fix, the best fix is the recent thumbnail fix! Thumbnails now have extra logic around generating them and now have a higher chance of actually being created! Let us know if you think there has been a change over the past month-ish.

Budget & Upcoming Migration

Reddthat has been lucky to have such a great community that has helped us stay online for over a year and if you can believe it, in just a few more months it will be 2 years, if we can make it.

Our costs have slowly increased over the years as you can all see by our transactions on OpenCollective (https://opencollective.com/reddthat). We've managed to reduce some costs in our S3 hosting after it balooned out and bring it down to a more manageable level. Unfortunately as well, the current economic issues have resulted in the Australian dollar slipping further and as we pay everything in USD or EUR it has resulted in slightly higher costs on a month-to-month basis..

Our best opportunity to keep online for the foreseeable future is to downsize our big server from a 32GB ram instance to a 16GB ram instance which will still provide enough memory that we will be able to function as we currently do while not affecting us in a meaningful way.

This means we'll need to reassess if running all our different front ends are useful, or do we only choose a few? Currently I am looking to turn off next and alexandrite. If you are a regular user of these frontends and prefer them please let me know as from our logs these are the least used while also take up the most resources. (Next still has bugs regarding caching every single image).

We can get a vps for about ~A$60-70 per month which will allow us to still be as fast as we are now while saving 40% off our monthly costs. This will bring us to nearly 90% funded by the community. We'll still be slowly "losing" money from our open collective backlog but we'll have at least another 6 months under our belt, if not 12 months! (S3 costs and other currency conversion not withstanding).

All of this will happen in late March early April as we will need to make sure we do it before the current contract is up so we don't get billed for the next month. Probably the 29th/30th of I don't fall asleep too early on those days.
It'll probably take around 45mins to 60mins but if I get unlucky maybe 2 hours.

Age Restriction

Effective immediately everyone on Reddthat needs to be 18 years old and futher interaction on the platform confirms you are over the age of 18 and agree with these terms.

If you are under the age of 18 you will need to delete your account under Settings

This has also been outlined in our signup form that has been updated around the start of February.

Australian & UK Policy Changes

It seems the UK has also created their own Online Safety Act that makes it nearly impossible for any non-corporation to host a website with user generated content (USG). This is slightly different to the Australian version where it specifically targets Social Media websites.

Help?

I would also like your help!
To keep Reddthat online, and to help comply with these laws, if you see content or user accounts which are under the age of 18 please report the account/post/content citing that the user might be under the age of 18.
We will then investigate and take action if required.

Thanks everyone

As always keep being awesome and having constructive conversations!

Cheers,

Tiff!

PS. Like what we are doing? Keep us online for another year by donating via our OpenCollective or other methods (crypto, etc) via here

Let's bring in the new year with some maintenance.

We are going to update Lemmy to the latest version 0.19.8

To perform this there will be a downtime of 30 to 45 minutes while our database updates.

Unless everything goes wrong we'll be up within the hour.

See you on the other side.

Tiff

Edit 1:
We are all done!

Edit 2:
Frontends didn't start correctly and should be sorted now

Hello Reddthat-thians!

As always here is our semi-whenever there is news update. I would, as always like to thank you all for being here and for the kind support we received last time I made an update.

We hit a few couple of milestones this last quarter:

• Our first BTC transaction was received! Thank you anonymous!
• Lemmy released two versions (which we have yet to update too, more below)
• I only restarted the services once the whole quarter as I thought we were down/stuck.

Lemmy 0.19.7

The latest update brings some fixes to Lemmy as well as ~~new features such as Private/Invite only communities. I can't wait to see what this does to help people find safe spaces and to self regulate.~~ Edit: that's in 0.20.0. The only new features (worth talking about imo) are parallel sending and allowing people to have 1000 characters in their bio.
This update is not live yet as the update requires a 30-60 minute database update that I want to test on a backup to make sure we can safely update. Other big instances have already done the update and most things went successfully so I'm confident that we can update without much fuss but I'd like to actually enjoy my holidays rather than spending 4 hours debugging db migrations! So, testing first.

Users

Something I think we struggle with is our proposition. As we have an opinionated view on what our community should be which doesn't always resonate when trying to sign up. With "only" 100 users active daily compared to other servers with over 1k it puts into perspective that not everyone agrees with our policies, but there are certainly people who do!

As such I'm open to ideas on growing our userbase or ideas on rephrasing our signup page, and sidebar.

Australian Social Media Laws

Here is the biggest news. Last week the Australian Government passed a new law that requires Social Media to no longer be accessible for kids under the age of 16. As such we will need to design a system & possibly modify Lemmy to be compliant with these laws.

I will not debate whether this is good or bad, because at the end of the day, we only have a few options: comply with these laws, don't comply and turn off reddthat, sell / handoff to another admin.
This has been a fun side project for me and I want to continue it. As such I'm looking into ways to comply with the local laws to ensure that Australian minors are blocked.

For Reddthat, this law requires social media to take "reasonable" steps to ensure children under the age of 16 are not be given an account. They do not define what reasonable means and leave it up to the social media platforms to define it. Really helpful for those indie social media platforms...

Unfortunately, like myself, I was thinking that we might technically not be defined as a social media platform thus skirting around the rule. The law defines a social media platform as any site whose primary or significant purpose is to connect 2 or more users, allows end-users to link to or interact with other end-users, and that allows end-users to post content.
So Lemmy, Mastodon, Blusky et al will be affected.

Researching it more leads to the idea which puts the onus on social media companies to continually verify accounts, so they take a proactive step to verify accounts that are believed to be accessed by minors under 16. Thus you need to prove you are above 16 years old. The cherry on top? You are not allowed to use a government issued ID to do it.
Thinking back to when you were 16 years old, what forms of identification did you have? A driver's licence? Maybe a passport? A school ID card? The last one (school ID) would be the only valid id you could use to verify yourself that isn't government issued in this instance. And good luck if you don't go to school anymore and started in the workforce when your 16.

Needless to say this is going to be a recurring theme over the next year and I will keep you all informed about it with our updates.

Future Features

A general outline of what I'm hoping to achieve is more controls in Lemmy to:

• have accounts in a 'monitored' state
• have a way to customise approval processes
• have accounts use a trust rating based on age, post/comment numbers, etc.
  • this would help with spam as well (obviously this would need a lot of factors)
• have automod built in (or actually setup correctly for us)

Obviously these are things that will need Lemmy development to help facilitate and I'll be creating a few issues over the weeks once I've fleshed out what a solution might look like.
If you have ideas please share! let's start the conversation on what the processes would look like to help solve our issues.

Financials

Thanks to the big donation drive from last time I posted we are still looking healthy enough in the financials to last until May next year! Which is good as our server plan renews in April. At that time we will be downgrading to a smaller instance as we may have over exaggerated when purchasing last April. We'll obviously know more closer to April next year about our financials and what we can afford to ensure we are cost neutral (if possible!).

Our LemmyWorld proxy also is an extra cost that we never budgeted for and has been ticking away successfully. Maybe by April LW will have upgraded to >0.19.5 which will give us parallel sending allowing us to remove the proxy.

Our object storage is humming along but costs are creaping up. I'll be doing an audit and possible look into doing a cost analysis to see if other object storage solutions would be cheaper. But for less than $20/m it's probably not even worth my time...

As a reminder we have many ways to donate if you are able. A recurring donation of $5 is worth more to me than a once of $60 donation. Only because it allows me to forecast the year and work out when we need to do donation drives or relax knowing everything in it's current state will be fine.

• Open Collective (preferred)
• Kofi
• Litecoin Direct
• Monero Direct (preferred)

Note: while you still can transfer me Bitcoin, I have removed it as an option because of the current transaction fees. Monero or Litecoin offer transaction fees as low as $0.005 so they are the preferred options compared to the $5 transaction fee of Bitcoin.

Conclusion

You are awesome. Posting, Commenting, and interacting with communities on Reddthat or through Reddthat makes it enjoyable every time to write these updates. So keep being awesome, even if you are a lurker!

If I were not hosting in Australia I would still be required to conform to laws if I were to allow Australian people to use Reddthat. Blocking all of Aus would be an option. But we need to ask ourselves does that fall under our values or ideals? (No obviously). Neither does requiring people to verify they are over a certain age. But we'll see what's to become.
This is the same as how CPPA or GDPR are still enforcable while we are all the way on the other side of the world.
This doesn't require a knee jerk reaction but requires serious thinking, whiteboarding and well thought out communication.

As always,
Cheers,

Tiff

PS Happy Holidays!

We had a brief outage today due to the server running out of space.

I have been tracking our usage but associated it with extra logging and the extra build caches/etc that we've being doing.

Turns out the problem was the frontend Next-UI which has been caching every image since the container was created! All 75GB of cached data!

Once diagnosed it was a simple solution to fix. I'm yet to notify the project of this error/oversight and I'll edit this once Issues/PRs are created.
I also haven't looked at turning the caching off yet as my priority was recovering the main Reddthat service.

Thanks all for being here!

Tiff

Hello. It is I, Tiff. I am not dead contrary to my lack of Reddthat updates 😅 !

It's been a fun few months since our last update. We've been mainlining those beta releases, found critical performance issues before they made it into the wider Lemmyverse and helped the rest of the Lemmyverse update from Postgres 15 to 16 as part of the updates for Lemmy versions 0.19.4 and 0.19.5!

Thank-you to everyone who helped out in the matrix admin rooms as well as others who have made improvements which will allow us to streamline the setup for all future upgrades.

And a huge thank you to everyone who has stuck around as a Reddthat user too! Without you all this little corner of the world wouldn't have been possible. I havn't been as active as I should be for Reddthat, moderating, diagnosing issues and helping other admins has been taking the majority of my Reddthat allocated time. Creating these "monthly" updates should... be monthly at least! so I'll attempt start posting them monthly, even if nothing is really happening!

High CPU Usages / Long Load Times

Unfortunately you may have noticed some longer page load times with Reddthat, but we are not alone! These issues are with Lemmy as a whole! Since the 0.19.x releases many people have talked about Lemmy having an increase in CPU usage, and they have the monitoring to prove it too. On average there was a 20% increase and for those who have single user instance this was a significant increase. Especially when people were using a raspberry pi or some other small form factor device to run their instance.

This increase was one of the many reasons why our server migrations were required a couple months ago. There is good news believe it or not! We found the issue with the long page load times, and helped the developers find the solution! -

This change looks like it will be merged within the next couple days. Once we've done our own testing, we will backport the commit and start creating our own Lemmy 'version'. Any backporting will be met with scrutiny and I will only cherry-pick the bare minimum to ensure we never get into a situation where we are required to use the beta releases. Stability is one of my core beliefs and ensuring we all have a great time!

Donation Drive

We need some recurring donations!

We currently have $374.10 and our operating costs have slowly been creeping up over the course of the last few months. Especially with the current currency conversions. The current deficit is $74. Even with the amazing 12 current users we will run out of money in 5 months. That's January next year! We need another 15 users to donate $5/month and we'd be breaking even. That's 1 coffee a month.

If you are financially able please see the sidebar for donation options, go to our funding post , or go directly to our Open Collective and signup for recurring donations!

Our finances are viewable to all and you can see the latest expense here: https://opencollective.com/reddthat/expenses/213722

• OVH Server (Main Server) - $119.42 AUD
• Wasabi S3 (Image Hosting) - $16.85 AUD
• Scaleway Server (LemmyWorld Proxy) - $6.62 AUD

Scaleway

Unfortunately until Lemmy optimises their activity sending we still need a proxy in EU, and I havn't found any server that is cheaper than €3.99. If you know of something with 1GB RAM with an IPv4 thats less than that let me know. The good news is that Lemmy.ml is currently testing their new sending capabilities so it's possible that we will be able to eventually remove the server in the next year or so. The biggest cost in scaleway is actually the IPv4. The server itself is less than €1.50 so if lemmy.world had IPv6 we could in theory save €1.50/m. In saying all this, that saving per month is not a lot of money!

Wasabi

Wasabi S3 is also one of those interesting items where in theory it should only be USD$7, but in reality they are charging us closer to USD$11. They charge a premium for any storage that is deleted before 30 days, as they are meant to be an "archive" instead of a hot storage system.

This means that all images that are deleted before 30 days incur a cost. Over the last 30 days that has amounted to 305GB! So while we don't get charged for outbound traffic, we are still paying more than the USD$7 per month.

We've already tried setting the pictrs caching to auto-delete the thumbnails after 30 days rather than the default 7 days, but people still upload and delete files, and close our their accounts and delete everything. I expect this to happen and want people to be able to delete their content if they wish.

OVH Server

When I migrated the server in April we were having database issues. As such we purchased a server with more memory (ram) than the size of the database, which is the general idea when sizing a database. Memory: 32 GB. Unfortunately I was thinking on a purely technical level rather than what we could realistically afford and now we are paying the price. Literally. (I also forgot it was in USD not AUD :| )

Again, having the extra ram gives us the ability to host our other frontends, trial new features, and ensure we are going to be online incase there are other issues. Eventually we will also increase our Lemmy applications from 1 to 2 and this extra headroom will facilitate this.

Donate your CPU! (Trialing)

If you are unable to donate money that is okay too and here is another option! You can donate your CPU power instead to help us mine crypto coins, anonymously! This is a new initiative we are trialing. We have setup a P2Pool at: https://donate.reddthat.com. More information about joining the mining pool can be found there. The general idea is: download a mining program, enter in our pool details, start mining, when our pool finds a "block", we'll get paid.

I've been testing this myself as an option of the past month as a "side hustle" on some laptops. Over the past 30 days I managed to make $5. Which is not terrible if we can scale it out. If it doesn't takes off, that's fine too!

I understand some people will be hesitant for any of the many reasons that exist against crypto, but any option to help us pay our server bills and allow people to donate in an anonymous way is always a boon for me.

Conclusion

These Lemmy bugs have been causing a headache for me in diagnosing and finding solutions. With the upcoming 0.19.6 release I hope that we can put this behind us and Reddthat will be better for it.

Again, thank you all for sticking around in our times of instabilities, posting, commenting and engaging in thoughtful communications.

You are all amazing.

Cheers,

Tiff

It's our Birthday! 🎂

It's been a wild ride over the past year. I still remember hearing about a federated platform that could be a user driven version of Reddit.

And how we have grown!

Thank you to everyone, old and new who has had an account here. I know we've had our ups and downs, slow servers, botched migrations, and finding out just because we are on the otherside of the world Lemmy can't handle it, but we are still here and making it work!

If I could go back and make the choice again. Honestly, I'd probably make the same choice. While it has been hectic, it has been enjoyable to the n'th degree.
I've made friends and learnt a lot. Not just on a technical level but a fair amount on a personal level too.

We have successfully made a community of over 300 people who regularly use Reddthat as their entryway into the Lemmyverse. Those numbers are real people, making real conversations.

Here's to another amazing year!

Tiff

PS: I'm still waiting for that first crypto donation 😜

PPS: Would anyone like a Hoodie with some sort of Reddthat logo/design? I know I would.