It seems like they are down for a longer time now. How will they recover? Does longer down mean they will have to do more catching up with other instances? Can I get updates somewhere?
Lemmy.ml did a server migration which was posted about 8 hours ago which was pinned on lemmy.ml meta. It appears in the comments that some users have experienced a 502 error, which was related to “iCloud’s Private Relay”. They were able to bypass the issue using a VPN, although I haven’t had any issues accessing lemmy.ml on my pc. I also read something about AWS us-east-1 being down, so it could also be related to that.
us-east-1 must not be completely down, or I’d be having a much worse day!
Saw it on https://lemmy.world/post/105778
Ah, their incident page says it was mostly lambda, my infra is mostly all on EKS/EC2.
Following communities could be another issue entirely, sometimes you have to subscribe then unsubscribe multiple times in order to get it to work.
@Sleeping I can’t subscribe in the 1st place. Guess I’ll just have to note down the community and try subscribing tomorrow again as it’s late around here 😁
It’s weird being part of broad communities like technology or games again. On reddit I unsubbed all the default subs and subscribed to more niche things like 3D printing or weird games. It’s quite a change to see people talking about things like mainstream AAA games. Intellectually I knew that a lot of people had to get hyped about the latest Bethesda game or Apple products or whatever, but I think part of me stopped believing it, because it still surprises me whenever I see it.
This appears to have somehow ended up in the wrong spot. I hadn’t even clicked on this thread before, and yet here’s my reply to a different one. Ah well.
Yeah, I’m a mod of retrogaming and a few few other subs in that network. Best group of users ever. 320k people in just the one sub and I think as a team we had to take down maybe 3-4 posts a day? Maybe? I banned 2-3 people in a month, that’s absolutely unheard of in a sub that size. I think the worst argument I saw the whole time was a running “the playstation one was not called the PSX” “Yes it was” “NO IT WASN’T!” “HERE IS AN ARTICLE, YES IT WAS.” thing that we had to specifically make a rule about. Odd little drama, but other than that? No fights at ALL. I’m talking a report maybe every few days for something being off topic if we didn’t get to it quick enough.
The niche subs about fringe interests, even if they’re popular ones, often bring out the very best people. And the patient/retro gamers are so very chill.
Not sure where they host but AWS is currently in the middle of a pretty big outage so it could be related.
In short, yes it will take longer to “catch up”.
The longer answer, be patient. This is an issue that will be resolved. I’m sure that the admins are doing their best to get the instance up & running ASAP.
From my experience during the great Twitter migration to Mastodon, you saw the same thing. Especially the “main instance(s)”, mastodon.online and mastodon.social in this case, were overloaded to the brim. After a day or two, the sysadmins and developers were able to stabilize the instance.
The sysadmins in this case were responsible to keep the servers up and running (and scale up where needed). The developers because their code has been “battle tested” for this level of interaction. So changes were needed, to optimize and squash bugs.
I hope this helps 😊
This is a fantastic point. I watched several migration waves from Twitter to the Mastodon/*oma/*key network over the course of Nov-Feb and each became less impactful as instances and developers understood better how that traffic would flow. Consider also that those platforms had also been around for a few years already.
Lemmy and kbin are very, very young still, so it will be great to see them develop over the course of the coming months. I expect the next couple weeks leading to the June 30 3PA closures on Reddit will be spent preparing for another wave.
Oh definitely that the next couple of weeks will be spent preparing for a new wave. I wouldn’t be surprised if that was the reason why Lemmy.ml was down. Yes I know that it was because of a server migration but, that server migration also didn’t happen for no reason ;-)
The first migration wave to Mastodon was very bad. I still remember, that following a person was nearly impossible. Not to mention the fact that, even if you could follow someone, it didn’t automatically meant their messages ended up in your feed.
I know for example that version 0.17.4 of Lemmy (that was released 2 days ago) already has some database optimizations.
