Hi all,

If you’re just now signing in for the first time in 12+ hours, you may just now be finding out that Lemmy World and other instances where hijacked. The hijackers had the full abilities of hijacked user, mod, and admin accounts. At this time, I am only aware of instance defacing and URL redirections to have been done by the hijackers.

If you were not forced to sign back in this morning, contact your instance admin to verify mitigations were completed on your instance.

How?

This occurred due to an XSS attack in the recently added custom emojis. Instance admins should follow the issue tracker on the LemmyNet GitHub, as well as the Matrix Chat. Post-Incident Activity is still on-going.

Currently, it is likely that just your session cookie was stolen, with instance admins being targeted specifically by checking for navAdmin, an HTML element only instance admins had. I do not believe this to affect users across instances, but I have yet to confirm this.

What happens next?

As I am not the developers or affected instance admins, I cannot make any guarantees. However, here is what you’ll likely see:

  1. Post Incident investigation continues. This will include inspecting code, posts, websites, and more used by the hijackers. An official incident writeup may occur. You should expect the following from that report:
  • Exactly what happened, when.
  • The incident response that occurred from instance admins
  • Information that might have helped resolve the issue sooner
  • Any issues that prevented successful resolution
  • What should have been done differently by admins
  • What should be improved by developers
  • What can be used to identify the next attack
  • What tools are needed to identify that information
  1. A CVE is created. This is an official alert of the issue, and notifies security experts (and enthusiasts), even those not using lemmy, about the issue.

  2. A code security audit is done. This will likely just be casual reviews by technical lemmy users. However, I will be reaching out to the Mozilla Foundation and Cure53 as they recently did an audit of Mastodon. If there is interest in an external audit of lemmy and the costs are affordable, I’ll look into crowdfunding this cost.

201 points
*

This incident made me realize not to use an admin account for my primary lemmy account in my personal instance. I setup another account for instance admin purpose (with 2FA enabled) and keep it logged out, then remove my primary account from the instance admin list.

permalink
report
reply
76 points

This is a good mindset in general, when working in AWS you are not supposed to use your root account unless it’s absolutely necessary even if you are the only user. Hosting a Lemmy instance should be no different.

permalink
report
parent
reply
11 points

Yeah, even in Windows or what-have-you, you should always keep your admin account separate from your daily driver account for exactly this reason.

permalink
report
parent
reply
6 points

most, if not every, linux distro work that way

permalink
report
parent
reply
2 points
*

I just setup a VPS for a Minecraft server for some friends and did exactly that. I was under a bit of a time crunch, but still took the time to think through those challenges in access for everything. Created an unprivileged user to run the server as, created a seperate unprivileged user for another service. Disabled password-based SSH login, etc.

I should probably setup a dedicated non-root admin account for administrative functions but that’s a problem for after work

permalink
report
parent
reply
17 points

I work with 2 factor, Oath, SAML, etc. all the time for work, and for the life of me I can’t get it working properly with Lemmy.

permalink
report
parent
reply
21 points

Lemmy decided to go with SHA256 for TOTP seed. This is a very odd move since many 2FA apps don’t support SHA256. I actually had to write a quick python script to spit out my 2FA code since Bitwarden doesn’t support it. Hopefully either Lemmy will change to SHA-1 or Bitwarden will start to support SHA256 seeds.

permalink
report
parent
reply
17 points
*

Wait, I use bitwarden and it works just fine. And yes, it’s using SHA256.

Screenshot:

permalink
report
parent
reply
12 points

SHA-1 has been broken already.

permalink
report
parent
reply
10 points

Preferably the latter, nobody should really be using SHA1 for anything security focused and new

permalink
report
parent
reply
7 points

I use bitwarden, so it’s pretty simple: just copy the link from the 2fa button in Lemmy into the TOTP field in your account editor in bitwarden, and it’ll automatically recognize the format.

permalink
report
parent
reply
1 point

Yeah, I’m using Google Auth, and I’m getting codes just fine, but none of them work, regardless of how I try to use it.

permalink
report
parent
reply
6 points

Yup. Basics of running a server for anything. Never use your admin account and make a default backup with 2 factor.

permalink
report
parent
reply
99 points

Having recently migrated from Reddit (and kept up with commercial social media hacks) I’m used to Nothing To See Here! We totally didn’t store your personal information in plaintext for hackers to snatch. Oh and maybe please change your passwords. All Part Of The Show!

So, by comparison, the response here is downright heartwarming.

permalink
report
reply
8 points

Whoops! Reddit spazzed out and couldn’t send your post because it hurts spez’s feelings!

permalink
report
parent
reply
69 points

If there is interest in an external audit of lemmy and the costs are affordable, I’ll look into crowdfunding this cost.

It could get VERY, VERY expensive… depends on code complexity.

permalink
report
reply
67 points
*

Not at this stage.
Lemmy grew too fast, got many more eyes.
Step 1 is getting a security focus group selected from the people who contribute code to lemmy.

Just like the admins and coders volonteer their time, security specialists will too, money might be needed, but that is not in the the first steps.

permalink
report
parent
reply
14 points

Agreed, this is wise.

permalink
report
parent
reply
24 points

Yeah, specifically why I mentioned “affordable”.

permalink
report
parent
reply
2 points

is there a mailing list or anything like that? i would like to help contribute to security hardening efforts

permalink
report
parent
reply
2 points

They have a Matrix chatroom I believe.

permalink
report
parent
reply
2 points

There’s this dev chatroom https://matrix.to/#/#lemmydev:matrix.org

permalink
report
parent
reply
11 points
  1. Set up a fake Nazi lemmy
  2. Have it online during the next Chaos Communication Congress
  3. ???
  4. 50 CVEs.
permalink
report
parent
reply
17 points
  1. Be Lemmy Devs
  2. Loudly Claim Lemmy is Bulletproof and unhackable
  3. ???
  4. 50 CVEs
permalink
report
parent
reply
3 points

This one’s better… and simpler 👍.

permalink
report
parent
reply
6 points

Agreed. It might be hard to swing right now, but imo this is going to be a crucial step moving forward.

permalink
report
parent
reply
1 point

The examples below might actually do most of the work for free.

permalink
report
parent
reply
46 points
*
Deleted by creator
permalink
report
reply
4 points

HTML injection / XSS vulnerabilities tend to be a sign of amateur hour to be honest. Made me a bit worried that I’m hitching my reddit escape wagon to the wrong technology. But sounds like this was due to instance-specific customizations rather than the core Lemmy tech, so hopefully we’re still on solid ground.

permalink
report
parent
reply
4 points

The way the hack was utilized is honest very creative and interesting;

That’s often the case with exploits.

permalink
report
parent
reply
6 points
*
Deleted by creator
permalink
report
parent
reply
0 points
*

Sure. Not sure how that’s relevant though?

In general, finding an exploit requires looking for little tiny details that could exist in, really, any area of a given system; looking for a bug, and then exploiting that bug by understanding how input data can be used to create a deterministic chain of events.

This almost always requires thinking outside of the box.

There are people who are also paid to find these before malicious actors do.

It’s always going to be creative in some way, at least in the beginning.

It’s like when people first discover Quake’s fast inverse square root. Sure, the first time around it seems genius. In reality, code like that is actually everywhere, and there is a somewhat trivial aspect to optimizing those kinds of problems.

permalink
report
parent
reply
0 points

No, Bobby Tables is really not that creative at all. It is the most basic, entry-level exploit.

permalink
report
parent
reply
27 points

Was lemmy.ml affected ?

permalink
report
reply
45 points

Only Lemmy instances with custom emoticons were affected based on the Recap of the Lemmy XSS incident. So if Lemmy.ml doesn’t have these it should not have been affected.

permalink
report
parent
reply
11 points

I think generally the less flashy features (pun intended) the better. Text and links (and well sanitized) is all we need.

permalink
report
parent
reply

I agree. Sadly it seems the rest of the world does not. Hopefully as Lemmy matures we can get to a point where features are not pushed put half-baked because there aren’t enough people willing and able to give thorough code reviews.

permalink
report
parent
reply
-2 points

Yeah. Hopefully the devs will learn to wait before pushing new technologies that don’t really matter. (they won’t)

permalink
report
parent
reply
5 points

Well, I’m glad I didn’t bother to add them.

permalink
report
parent
reply
-1 points

Less is more.

permalink
report
parent
reply

Fediverse

!fediverse@lemmy.world

Create post

A community to talk about the Fediverse and all it’s related services using ActivityPub (Mastodon, Lemmy, KBin, etc).

If you wanted to get help with moderating your own community then head over to !moderators@lemmy.world!

Rules

  • Posts must be on topic.
  • Be respectful of others.
  • Cite the sources used for graphs and other statistics.
  • Follow the general Lemmy.world rules.

Learn more at these websites: Join The Fediverse Wiki, Fediverse.info, Wikipedia Page, The Federation Info (Stats), FediDB (Stats), Sub Rehab (Reddit Migration), Search Lemmy

Community stats

  • 5.3K

    Monthly active users

  • 1.9K

    Posts

  • 65K

    Comments