this post was submitted on 05 Jul 2023
3369 points (99.4% liked)

Lemmy.World Announcements

30442 readers
4 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages πŸ”₯

https://status.lemmy.world/

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations πŸ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 2 years ago
MODERATORS
 

Another day, another update.

More troubleshooting was done today. What did we do:

  • Yesterday evening @phiresky@phiresky@lemmy.world did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
  • @cetra3@lemmy.ml created a docker image containing 3PR's: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
  • We started using this image, and saw a big drop in CPU usage and disk load.
  • We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a return 404 in nginx conf for /api/v3/ws.
  • We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
  • We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set ~~proxy_next_upstream timeout;~~ max_fails=5 in nginx.

Currently we're running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the ~~proxy_next_upstream timeout;~~ max_fails=5 workaround but for now it seems to hold with 1.

Thanks to @phiresky@lemmy.world , @cetra3@lemmy.ml , @stanford@discuss.as200950.com, @db0@lemmy.dbzer0.com , @jelloeater85@lemmy.world , @TragicNotCute@lemmy.world for their help!

And not to forget, thanks to @nutomic@lemmy.ml and @dessalines@lemmy.ml for their continuing hard work on Lemmy!

And thank you all for your patience, we'll keep working on it!

Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.

Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that's now started, and I noticed the proxy_next_upstream timeout setting didn't work (or I didn't set it properly) so I used max_fails=5 for each upstream, that does actually work.

(page 6) 50 comments
sorted by: hot top controversial new old
[–] bluemoose@lemmy.world 3 points 2 years ago

Thanks for all the work you've put into this!

[–] fubo@lemmy.world 3 points 2 years ago

Woo hoo, nice graphs! Monitoring data that shows big improvements is always fun to see.

[–] Richard@lemmy.world 3 points 2 years ago* (last edited 2 years ago) (2 children)

Minor thing but over night both wefwef and Memmy clients are showing the wrong comment score (karma) against my profile, and given they are showing the same amount I assume it’s related to API fed data. Value was correct yesterday. Easy for me to confirm given I have only 2 dozen posts and the value has dropped to single digits.

Not a biggie, but figured I’d report it in case there was some issue causing that. Might be some optimisation around indexing or something has intentionally or unintentionally impacted that.

Otherwise the service feels much more stable currently. No timeouts today where it’s been very frequent the past few days. Nice job. πŸ‘

load more comments (2 replies)
[–] Graphy@lemmy.world 2 points 2 years ago

I read this post after browsing for an hour and I’ve definitely felt a change. Great work!

[–] devopspalmer@lemmy.world 2 points 2 years ago

Amazing work @ruud@lemmy.world and admin team! Good to see some really awesome progress!

[–] carroarmato0@lemmy.world 2 points 2 years ago

Thanks for the tremendous work!

[–] GeneralStrike@lemmy.world 2 points 2 years ago

Absolutely great job! Everything feels buttery smooth today, and I’m yet to see any errors.

[–] cheeseblintzes@lemmy.world 2 points 2 years ago

Thank you guys so much for all you do.

[–] Woshimaraci@lemmy.world 2 points 2 years ago
[–] Zenbach@lemmy.world 2 points 2 years ago

Working great thanks for all you do!

[–] doryx@lemmy.world 2 points 2 years ago
[–] boeman@lemmy.world 2 points 2 years ago

Awesome work! I'm seeing a lot less errors and I can actively scroll through posts quickly now.

I'm in awe of how quickly these issues are being resolved.

Thank you for all the hard work.

[–] patachu@lemmy.world 2 points 2 years ago

Thank you to all the code monkeys for getting things straightened out!

System appears to be working as intended

[–] CataclysmZA@lemmy.world 2 points 2 years ago

Stuff is pretty smooth now!

Looks really good, and have definitely noticed less issues just over the last couple of hours. Great job folks!

[–] FlaxPicker@lemmy.world 2 points 2 years ago (4 children)

Not that it really matters, but did anyone else notice their comment/post points get deleted after this update? Seems odd.

load more comments (4 replies)
[–] Anaralah_Belore223@lemmy.world 2 points 2 years ago* (last edited 2 years ago)

Using anaralah_belore223@lemmy.world

Commenting took about 3 seconds

Editing took about 0.8-2 seconds

(up/down)voting took about 1 second

[–] wit@lemmy.world 2 points 2 years ago

Are you guys able to create reports? I am not.. It keeps spinning.

[–] Desistance@lemmy.world 2 points 2 years ago* (last edited 2 years ago)

It's definitely snappier than before. Constantly having to refresh got old quick.

[–] burgundymyr@lemmy.world 2 points 2 years ago

This is a lot of work, I genuinely feel honored by the service of everyone involved. Thank you!

[–] Narann@lemmy.world 2 points 2 years ago

I can't imagine the amount of work Lemmy's devs and ITs are under since few days, but those are important for the future of Lemmy. Keep the good work! You're awesome!

[–] dq9@lemmy.world 2 points 2 years ago

Thank you so much for all your hard work!

[–] decadentrebel@lemmy.world 2 points 2 years ago

Awesome job as always, @ruud@lemmy.world

But these grey fonts on the sidebar will take some getting used to.

[–] HybridSarcasm@lemmy.world 2 points 2 years ago

Wow! Your commitment and diligence is admirable!

[–] CupDock@lemmy.world 2 points 2 years ago

So much smoother! Thank you for all your charity and hard work!

[–] CrypticCoffee@lemm.ee 2 points 2 years ago

That is so hot!

Thanks for sharing :).

Loving the transparency.

[–] wanderingmagus@lemmy.world 2 points 2 years ago

Thanks for all your hard work. The change is absolutely noticeable.

[–] Beardliest@lemmy.world 2 points 2 years ago

Really appreciate all the hard work going on behind the scenes! Feels night and day different after the changes. Also appreciate the transparency. Nice to see in this day and age.

[–] SteveJobs@lemmy.world 2 points 2 years ago

Thanks for the hard work!

load more comments
view more: β€Ή prev next β€Ί