this post was submitted on 10 Jul 2023
34 points (100.0% liked)

Beehaw Support

2851 readers
19 users here now

Support and meta community for Beehaw. Ask your questions about the community, technical issues, and other such things here.

A brief FAQ for lurkers and new users can be found here.

Our September 2024 financial update is here.

For a refresher on our philosophy, see also What is Beehaw?, The spirit of the rules, and Beehaw is a Community

This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

if you can see this, it's up  

founded 3 years ago
MODERATORS
Zmodem@beehaw.org
Gaywallet@beehaw.org
alyaza@beehaw.org
Lionir@beehaw.org
Penguincoder@beehaw.org
remington@beehaw.org
34
PSA: We're back, here's a post-mortem! (beehaw.org)
submitted 2 years ago* (last edited 2 years ago) by Lionir@beehaw.org to c/support@beehaw.org
51 comments fedilink hide all child comments
 

Hi Beeple!

Here's a vague version of events :

  • 11PM EST: Lemmy.world got hacked

  • 12:20AM EST: Blahaj.zone got hacked

  • 12:25AM EST: I shut down the server

  • 12:30AM EST: I make announcements to tell people about this

  • 12:45AM EST: I have an idea of what the problem is but there is no fix

  • 2:20AM EST: I go to sleep

  • 8:50AM EST: The server is booted back up, steps are applied to mitigate issues (Rotating JWTs, Clearing DB of the source of vulnerability, deleting custom emoji), UI is updated with the fix, CSP and other security options are applied

  • 11:40AM EST: We start testing things to make sure are working And well, now here we are.

If you have issues logging in or using an app:

  1. Log out if you somehow are still logged in

  2. Clear all cache, site data, etc.

  3. Hard refresh Beehaw using CTRL+F5

  4. Log back in.

If you still have issues, write to us at support@beehaw.org

To be clear : We have not been hacked as far as we know, we were completely unaffected. This was done preemptively.

Oh yeah, in case, you haven't, this is a good opportunity and reminder to follow us on Mastodon as the communication line was still up despite Beehaw being down : https://hachyderm.io/@beehaw

top 50 comments
sorted by: hot top controversial new old
[–] Hirom@beehaw.org 10 points 2 years ago (2 children)

The shutdown is a good call given the circumstances.

An idea of less-radical preventive action is placing the instance in read-only mode, either as a Lemmy feature, or through reverse proxy settings (eg reply 503 for any POST/PUT/DELETE request). But that'd require some development and/or preparation.

Doing that on the reserve proxy side would block any user-submitted content and more (logins, searches, ...). This would hopefully be efficient at blocking many attack vectors, while still keeping the instance partially online, even if that's a degraded mode.

[–] Lionir@beehaw.org 8 points 2 years ago

Note that if this were a Lemmy feature, if we had been infected, an admin could've gotten hacked and as a result, disabled that feature. I'm not really sure what can be done to make Beehaw foolproof. That said, the UI has since been hardened by CSP headers so this type of attack should no longer be possible.

[–] interolivary@beehaw.org 6 points 2 years ago

Would read-only mode help with XSS exploits though, like this particular one? Since the "damage was already done" by the time anybody noticed, wouldn't putting the site in read-only mode still have kept serving up the XSS payload? It'd stop "infected" people from making any state mutations on Lemmy, but eg. data exliftration would still happen

[–] communication@beehaw.org 3 points 2 years ago

Huge props for being one of the few major instances to preemptively shut down!

[–] mlburgess@beehaw.org 3 points 2 years ago

Glad it's back up. I went outside. It was hot af and boring.

[–] LoneLee@beehaw.org 2 points 2 years ago

This is why I am on Beehaw. The Admins really care about the Instance and the content on it.

That's why I want to bring attention to the fact, that U can support them. https://opencollective.com/beehaw

I am not a Admin, Mod or anything else. I just really like Beehaw and support them. And you should too.

[–] kittenroar@beehaw.org 2 points 2 years ago

Awesome work sidestepping the hack.

[–] GameGod@beehaw.org 1 points 2 years ago (1 children)

Content-Security-Policy will really help save your ~~bacon~~ beans and protect against XSS. Hopefully the Lemmy devs can apply a super strict policy to help. IMHO it's a must for any site with user generated content.

[–] Lionir@beehaw.org 2 points 2 years ago

This is what this PR has done as I understand it : https://github.com/LemmyNet/lemmy-ui/pull/1907

[–] frogman@beehaw.org 1 points 2 years ago

shutting down the server early was best. the nature of open source software is what allows these incidents to be mitigated as quickly as they are. thanks a lot to you guys, and to all of the team at Lemmy who worked to resolve this.

heroes <3

[–] emma@beehaw.org 1 points 2 years ago

morning thought: I've definitely joined the right instance. (also the start from the assumption of good faith guidelines linked to in Gaywallet's recent post)

[–] Pepper@beehaw.org 0 points 2 years ago (1 children)

12:30AM EST: I make announcements to tell people about this

I think it'd be beneficial to have more backup lines of communication for announcements than just Mastodon.

[–] Lionir@beehaw.org 2 points 2 years ago (2 children)

We have Discord and Matrix channels as well. Do you have anything to suggest?

[–] Pepper@beehaw.org 0 points 2 years ago (2 children)

Just something Google-friendly.

[–] Penguincoder@beehaw.org 2 points 2 years ago (1 children)

Nah.

[–] Pepper@beehaw.org 0 points 2 years ago (1 children)

I'll be blunt and say that unless you were already in-the-know, Beehaw pretty much ceased to exist when the server was shut down. Not the best result amidst a hacking scare.

[–] Penguincoder@beehaw.org 1 points 2 years ago (1 children)

Much preferable to the announcement of Beehaw was hacked and lost your user credentials . Security trumps convenience.

[–] Pepper@beehaw.org 0 points 2 years ago (1 children)

Having an entirely separate website, blog, or social media account for announcements that's accessible via a Google search wouldn't factor into how secure Beehaw is.

[–] Penguincoder@beehaw.org 0 points 2 years ago (1 children)
[–] Pepper@beehaw.org 0 points 2 years ago (1 children)

And how were users supposed to be able to see the sidebar while the server was offline?

[–] retronautickz@beehaw.org 1 points 2 years ago (1 children)

You could have checked it before and follow their Mastodon-style account and join their matrix and/or discord groups, like most of us did.

Because everything they do server-wise is announced in those places, preemptive shutdowns included.

Alternative ways to reach the admin team and to be kept aware of anything happening with the server exist. If you didn't take the time (seconds) to join at least one of them, that's not the server's owners fault.

[–] Pepper@beehaw.org 0 points 2 years ago (1 children)

Like most of us did

Considering the responses to the thread, I don't think that's true.

Alternative ways to reach the admin team and to be kept aware of anything happening with the server exist.

A lot of people, myself included, are still getting used to Lemmy. The status quo has been if stuff was happening to Reddit there was an easily accessible server status page you could search up. I tried to do the same this time around and Google came up with diddly-squat. I don't think googling Beehaw to figure out what's going on is that illogical of a response.

[–] retronautickz@beehaw.org 1 points 2 years ago* (last edited 2 years ago) (1 children)

Considering the responses to the thread, I don’t think that’s true.

Given that you aren't in any of the groups, nor following their mastodon account. I don't think what you think based on the responses on a sole thread has any merit

A lot of people, myself included, are still getting used to Lemmy. The status quo has been if stuff was happening to Reddit there was an easily accessible server status page you could search up. I tried to do the same this time around and Google came up with diddly-squat. I don’t think googling Beehaw to figure out what’s going on is that illogical of a response.

Every time you open Beehaw (or any Lemmy instance for the matter) on the right side you'll find the description of the server below the "trending communities" box. There the admins put important links, including the three alternative ways to reach them (Mastodon, Matrix and Discord). You didn't have to google anything, it was just taking a look the main page of the server, see the section titled "Beehaw" and read it (Something that could have been done any time the server was up)

[–] Pepper@beehaw.org 0 points 2 years ago (1 children)

Given that you aren't in any of the groups, nor following their mastodon account. I don't think what you think based on the responses on a sole thread has any merit

I'm gonna say this only once, but that's an incredibly rude thing to say. I was giving you the benefit of the doubt in my previous reply but it's obvious now that you're just treating me like an idiot.

I think what I've had to say has merit. Given the upvotes there are obviously some others following the conversation that share my view. Even if there wasn't though, I'd still think it's important that I spoke up.

[–] retronautickz@beehaw.org 2 points 2 years ago (1 children)

I’m gonna say this only once, but that’s an incredibly rude thing to say. I was giving you the benefit of the doubt in my previous reply but it’s obvious now that you’re just treating me like an idiot.

Yeah, because you haven't been rude from the beginning (you were)

People (including me) explained to you how you could have reached the admins and stay in touch so this kind of thing wouldn't take you by surprise. When you had been explained several times this things and you keep insisting you don't have any option because "you couldn't google it", one has to begin to think that you're doing this in bad faith.

[–] Pepper@beehaw.org 1 points 2 years ago (1 children)

I'm not going to apologise for stating that there should have been a page I could have reached via a Google search.

One has to begin to think that you're doing this in bad faith

I said you were rude because you started talking down to and attacking me directly as a person. That's not ok.

[–] retronautickz@beehaw.org 3 points 2 years ago (1 children)

You acted as if there weren't ways of reaching, only because it wasn't the one you wanted to have.

I suggest you to stop relying on google, because in general it doesn't give good result for Lemmy (less for Beehaw in specific), and start to follow/join at least one of the alternative groups/accounts

[–] Gaywallet@beehaw.org 3 points 2 years ago

This has devolved into a back and forth argument, lets kill the conversation here please. I think you've both made your point.

[–] Lionir@beehaw.org 0 points 2 years ago (5 children)

Can you be more precise? What exactly do you recommend? I don't know what would be more "Google-friendly"

load more comments (5 replies)
[–] gifflen@beehaw.org 0 points 2 years ago* (last edited 2 years ago) (1 children)

Something like status-page is always nice. I haven't used it but it looks like https://cachethq.io/ could be a decent fit as well.

[–] Lionir@beehaw.org 4 points 2 years ago (1 children)
[–] gifflen@beehaw.org 3 points 2 years ago

Heck yeah! Thanks for getting this up

[–] comicallycluttered@beehaw.org 0 points 2 years ago (1 children)

Agree with everyone else. Thanks for shutting it down.

I'll most likely do it anyway, but do you think password changes are necessary at this point?

[–] Lionir@beehaw.org 0 points 2 years ago (2 children)

I don't think this is necessary.

We had no messages on our database that had the vulnerability though some were federated from blahaj in the aftermath. The JWT, which is your session token, was changed as well so it seems very unlikely to me that this needs to be changed. There's no reason to believe the attack could've given access to passwords.

[–] abhibeckert@beehaw.org 1 points 2 years ago* (last edited 2 years ago) (1 children)

I don’t think this is necessary.

I'd add that it's basically useless. From what I've seen, resetting your password doesn't even invalidate previously issued JWT tokens, which would be the only reason to do it. But of course, you've already reset them all and so has lemmy.world.

[–] mainfrog@beehaw.org 1 points 2 years ago

A password reset probably should invalidate all previous JWT tokens.

[–] jarfil@beehaw.org 0 points 2 years ago* (last edited 2 years ago) (1 children)

We had no messages on our database that had the vulnerability

This is interesting. I actually commented about the use of emojis/emotes a couple days ago on a post on !foss@beehaw.org made by a federated user from lemmy.one, that has since been removed (😕), but I still have the bookmarked comment in which I copied the raw embed for the remote emote image in the federated comment I was responding to.

Do I understand it correctly, that the latest fixes to stop the code injection, will still allow remote image embedding, so something like an "emote picker extension to embed animated GIFs from a remote server and/or remote instance's emoji list" would still be doable and wouldn't pose any risk?

Or would such picker still have to include measures to prevent offering embeds with potential exploits?

[–] Lionir@beehaw.org 0 points 2 years ago (1 children)

Remote image embedding is not the issue, remote custom emojis would not have been an issue either. The issue, from my understanding, is that the way local emojis are rendered allowed for an XSS exploit.

You can look at the PR which fixed this issue if you have a better understanding of these things than me : https://github.com/LemmyNet/lemmy-ui/pull/1897/

I believe such a picker would be fine.

[–] jarfil@beehaw.org 0 points 2 years ago* (last edited 2 years ago) (1 children)

I see, so the prior emoji handling rendered content directly from the comment, instead of making sure it was strictly what was defined for the local emoji; that was a weird choice. Now they've also added a sanitizer wrapper to all of it in: https://github.com/LemmyNet/lemmy-ui/pull/1906

I guess the only downside of a picker that used the non-emoji image renderer, would be the loss of emoji CSS formatting.

[–] tanglisha@beehaw.org 1 points 2 years ago (1 children)

From what I can tell the whole point to the css class/formatting was controlling the size of the emojis. Depending on where they came from, I could see some being of random size and shape. Admins might not have the time or know-how to shrink them down, so css seems like a reasonable compromise as long as the files aren't huge.

I'm kind of bothered that the only fix seems to be on the frontend. Unfortunately, I haven't been able to stick with Rust long enough to take a reasonable crack at figuring out how to help on the backend. Input and output sanitization should ideally be handled in both places.

[–] jarfil@beehaw.org 2 points 2 years ago (1 children)

Lemmy's backend is kind of curious, in that it does the bare minimum to move content around and add some metadata fields.

For example, did you know that "deleting" a comment, only marks a "deleted: true" field, while the comment is still pushed in full to the frontend? Same thing happens with banned/mod removed comments, they just get marked as "removed: true" but otherwise still get pushed to the client, and the user can still edit them.

All the display processing is done in the frontend, or whichever app you happen to use.

[–] tanglisha@beehaw.org 0 points 2 years ago (1 children)

I can maybe see marking it as deleted in case someone wants to creat undelete functionality later. I don't agree with it, but I can see why someone would do it.

It's just weird to still push it to the frontend.

Same with the removed stuff. All of that should be handled on the backend and never even sent to the frontend. Sometimes the reason for deletion is something you don't want getting grabbed by someone who is bored and poking around in developer tools, like doxxing information.

Since I don't have the time to do anything about it, though, I guess I don't have a place to complain. I have strong feelings about this stuff, but there's a limit to the number of things a single person can work on. If I were to hop on an open source project this minute, it would be helping migrate Cursorless to an LSP.

[–] jarfil@beehaw.org 1 points 2 years ago* (last edited 2 years ago) (1 children)

There is an undelete feature, at least the Liftoff app has it.

Right now, the only effective way to delete a comment, is to edit and blank it, then you can delete it (well, the content, you still get shown as the author). The bright side is that even when the comment gets mod-deleted, you can still edit it... and everyone can still read it in the JSON... wait 🤔

someone who is bored and poking around in developer tools

No need, just view source. In the Liftoff app there is a "nerd stuff" option where you can also read the content.

Anyway, once a comment gets federated, it's out of the original user's control, can't really take it back.

[–] tanglisha@beehaw.org 1 points 2 years ago (1 children)

Hey, I wanted to thank you for this exchange. I learned a lot about lemmy.

[–] jarfil@beehaw.org 2 points 2 years ago

Glad to be of assistance. There's plenty of it I still haven't looked into, though.

load more comments
view more: next ›