this post was submitted on 10 Jul 2023
880 points (99.3% liked)

Lemmy.World Announcements

30241 readers
2 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages πŸ”₯

https://status.lemmy.world/

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations πŸ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 2 years ago
MODERATORS
 

While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.

As I am told, this was the issue:

  • There is an vulnerability which was exploited
  • Several people had their JWT cookies leaked, including at least one admin
  • Attackers started changing site settings and posting fake announcements etc

Our mitigations:

  • We removed the vulnerability
  • Deleted all comments and private messages that contained the exploit
  • Rotated JWT secret which invalidated all existing cookies

The vulnerability will be fixed by the Lemmy devs.

Details of the vulnerability are here

Many thanks for all that helped, and sorry for any inconvenience caused!

Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been 'stolen' and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).

For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.

top 50 comments
sorted by: hot top controversial new old
[–] LuckyLu@lemmy.world 184 points 2 years ago (12 children)

Very impressed by how quickly action has been taken by this and other instances to patch the issue.

[–] trouser_mouse@lemmy.world 26 points 2 years ago

Very, seems like great work.

load more comments (11 replies)
[–] ThisIsMyLemmyLogin@lemmy.world 83 points 2 years ago (18 children)

I wish hackers would invest their time in clearing credit card debt, deleting hospital fees, or something else that actually serves the public good, instead of hacking ordinary people just trying to get by.

[–] 00Xero00@lemmy.world 7 points 2 years ago (14 children)

I bet that little cunt Steve Huffman paid them to sabotage us

load more comments (14 replies)
load more comments (17 replies)
[–] TheVampireSaga@lemmy.world 65 points 2 years ago (1 children)

what steps are being taken to ensure it doesn't happen again? was any personal data compromised for users?

[–] ruud@lemmy.world 59 points 2 years ago (4 children)

Good point, I'll update the post.

[–] TheVampireSaga@lemmy.world 33 points 2 years ago (2 children)

Also I am curious, what's the easiest way to currently reach the admins in case this happens again somehow? Two of them on their account have been seemingly inactive for a month and as per your own statement you rarely check your notifications and dms. Is there a discord somewhere for it?

[–] ruud@lemmy.world 37 points 2 years ago (3 children)
[–] sirnak@lemmy.world 9 points 2 years ago (1 children)

Why wasn't there an info on /lemmy-world.statuspage.io ?

load more comments (1 replies)
load more comments (2 replies)
load more comments (1 replies)
[–] BustedPancake@lemmy.world 15 points 2 years ago (6 children)

So all our cookies are negated now with the JWT changed, and we just needed to login again? Can attackers have stolen our cookies in order to use our accounts to post as if it was us? I'm sure they were only interested in admin cookies, so most others were "useless" to them? I see nothing wrong with my posts so I should be safe, right?

[–] cantevencode@lemmy.world 17 points 2 years ago

Prior to the JWT secret being rotated, yes, they could have authenticated as you. The tokens are now all invalid and useless

load more comments (5 replies)
[–] InverseParallax@lemmy.world 9 points 2 years ago

Nice work on the recovery, especially from a 0-day.

[–] trouser_mouse@lemmy.world 52 points 2 years ago* (last edited 2 years ago) (6 children)

First - really good summary and sounds like everyone is working hard.

Cross posting the below comment.

Under GDPR if you have had a data breach you have a legal obligation to assess whether you need to report it and you must make the report within 72 hours of discovering the breach.

There are other types of reportable breaches too, I only mention data as it sounds most likely. You may or may not be subject to PECR which may also have been breached although less likely. I don’t really have enough familiarity with the regulation to discuss that one.

If you are not sure if there has been a breach you may also need to discuss it with the relevant body or make a report.

Please can you update what action you have taken regarding this and if the incident was reportable or not and the reasons why. Edit - from that new information, it sounds like this is a reportable breach.

For a full understanding, it would be good to know if you had 2FA enabled on the compromised account particularly as it had admin privileges and if so how 2FA was circumvented with this exploit.

It would also be good to know what measures you have in place to prevent the same or other malicious attempts on your Open Collective and Patreon accounts as issues with those are potentially more serious. They may not be vulnerable to this, but it is going to be reassuring to know there is good security practice, 2FA protection etc enabled and you have robust procedures in place.

[–] ruud@lemmy.world 26 points 2 years ago

Thanks for the info. We're looking into this.

[–] hawkwind@lemmy.management 10 points 2 years ago (2 children)

Out of curiosity, where would the regulators go for a case like this? There's no "company" running it per. se.

[–] linearchaos@lemmy.world 14 points 2 years ago

It seems the general consensus is GDPR applies even to OSS non company entities, but it would appear that there's very little being done to honor it.

https://www.zwilnik.com/better-social-media/activitypub-conference-2019/oss-compliance-with-privacy-by-default-and-design/#:~:text=Although%20GDPR%20directly%20applies%20only,sysadmins%2C%20including%20in%20the%20Fediverse.

This article outlines Fediverse and responsibilities, I think it mostly requires someone to file a lawsuit before there's any action.

In another case a man had cameras in his back yard that could also see a public area and was fined and forced to move them.

https://www.termsfeed.com/blog/gdpr-exemptions/

Mainly it just seems to be fodder to be used in lawsuits to make people comply with others security wishes. Not certain how all that works since cities are covered in public cameras.

load more comments (1 replies)
load more comments (4 replies)
[–] alaxitoo@lemmy.world 52 points 2 years ago

Thank you for your work πŸ™

[–] nosut@lemmy.world 33 points 2 years ago* (last edited 2 years ago)

Thanks for the work. As a heads up it appears most of the block instances are back however I believe explodingheads is still missing which you may want to confirm.

EDIT: it has been added back to the block list.

[–] AlmightySnoo@lemmy.world 31 points 2 years ago* (last edited 2 years ago) (1 children)

the details of the vulnerability are already known now anyway since there's a fix that was proposed on the Lemmy GitHub so I don't think it will hurt others to talk about it

[–] namelivia@lemmy.world 11 points 2 years ago (1 children)

Could you please link the issue? Thanks!

[–] namelivia@lemmy.world 24 points 2 years ago (2 children)
[–] AlmightySnoo@lemmy.world 25 points 2 years ago (8 children)

yup that's the one

what I find weird is that the "fix" still focuses only on the front-end, the issue is still that unescaped HTML is being stored in the database and still trusting the front-end is nuts

load more comments (8 replies)
load more comments (1 replies)
[–] bluemellophone@lemmy.world 29 points 2 years ago* (last edited 2 years ago) (1 children)

Can we get another admin to sign off on this being authentic? In other words, short of a signed GPG signature how do we trust announcements after a breach where admin accounts are compromised?

[–] ruud@lemmy.world 39 points 2 years ago (1 children)

Good point. I did post about this on Mastodon @mwadmin@mastodon.world

[–] bluemellophone@lemmy.world 9 points 2 years ago (1 children)
[–] hawkwind@lemmy.management 11 points 2 years ago (3 children)

Don't fall for it. They're also an admin on mastodon.world! :)

load more comments (3 replies)
[–] brittleback@lemmy.world 22 points 2 years ago

Well done all involved. Sounds like it was caught and mitigated quickly

[–] wazoobonkerbrain@lemmy.world 18 points 2 years ago (2 children)

IMPORTANT ANNOUNCEMENT: My account was not among those hacked. Any random bullshit appearing in my post/comment history was written by me.

load more comments (2 replies)
[–] PagingDoctorLove@lemmy.world 15 points 2 years ago (2 children)

Can I ask some possibly dumb questions?

  1. What is JWT?
  2. Was any private user data compromised, and if so will users be informed?
  3. Is there anything regular users can do to avoid their data being compromised? For example, not accessing lemmy on certain web browsers?

Thank you!

load more comments (2 replies)
[–] dr_scientist@lemmy.world 15 points 2 years ago

Good job. I don't understand very much of that, so that makes me all the more grateful. Thank you.

[–] MeshPotato@lemmy.world 13 points 2 years ago

Thanks for your efforts. I know that Lemmy was put in place rather quickly as a Reddit alternative. But I'm genuinely hopeful that this will be a good alternative.

[–] jarfil@lemmy.ml 12 points 2 years ago (8 children)

How does this impact those using mobile apps like Jerboa or Liftoff, instead of the website directly?

load more comments (8 replies)
[–] AlmightySnoo@lemmy.world 10 points 2 years ago (1 children)

One thing I don't get. Custom emojis can only be created by an admin, but you're saying an admin's account here got compromised because of that and not the other way around. Does that mean that an evil instance set a custom emoji with the injected JavaScript and propagated it to the federated instances?

[–] Calyhre@lemmy.world 7 points 2 years ago* (last edited 2 years ago)

From the fix, I believe the custom emojis were not double checked after a user submits a post. The post data was used to display the emojis, and thus allowing injection.

The fix now is to search the emojis in the custom emojis list from the backend rather than the user post.

[–] cantevencode@lemmy.world 9 points 2 years ago (2 children)

Does an admin account have any permissions to view email addresses or data of registered users?

Did MichelleG not have 2FA enabled?

Now that this has happened, it's be worth pushing this issue through as high priority. If HttpOnly was enabled, then an admin takeover would not have been possible.

https://github.com/LemmyNet/lemmy-ui/issues/1252

[–] nosut@lemmy.world 21 points 2 years ago (1 children)

The JWT exploit bypasses 2FA requirements. It basically steals your active session and allows a third party to use it.

[–] cantevencode@lemmy.world 8 points 2 years ago (2 children)

Good point. I suppose the only way to fix that particular issue to disallow cookie authentications from a new location

load more comments (2 replies)
load more comments (1 replies)
[–] ulu_mulu@lemmy.world 8 points 2 years ago

Amazing how you quickly reacted to this!! Bravo!!

TIP: if you can't login after what happened, clear out your browser cache including ALL cookies, that fixes it (it did for me at least). I believe it's also advisable to change lemmy password.

[–] 00Xero00@lemmy.world 7 points 2 years ago* (last edited 2 years ago) (2 children)

I can't log into my account anymore, this one is a new one I've just made. I tried to reset my password but nothing came in the mailbox. I can still see comments and posts from that account though.

It's this one:

And I don't know why but I can't save the profile pic for this account.

Edit: Nvm, I use another email to sign up for Lemmy and forgot about it

load more comments (2 replies)
load more comments
view more: next β€Ί