Hey folks!

We unfortunately had about half an hour of unplanned downtime today. This was caused by an issue with our hosting provider. The issue is solved for now, and I am planning to make some changes to prevent similar issues in the future. Sorry for the inconvenience!


Technical details

Our servers are communicating with our database over Hetzner’s “vSwitch” service. Unfortunately, this service seems to be quite flaky - over the past few months, I have had to deal with the connection just dropping without recovering many times. Mostly this has not resulted in any noticeable downtime, as we have redundant servers, so even if one of them stops working, it won’t affect lemm.ee users. However, in this instance, all of our API servers lost their connection to our database at the same time, which resulted in actual downtime.

I have now decided to migrate our setup away from the vSwitch in the near future to hopefully stop these issues for good. Should be possible to do this migration without any downtime, I just need to set aside some time to actually create an alternative solution for us, most likely over the coming weekend. I will update this post once the migration is complete.

Update: the migration is now complete! You can read more here.

31 points

Thanks for the update, even with that little downage .ee is insanely stable!

permalink
report
reply
21 points

Thanks for the update. If you have the time and energy after the migration I’d be interested to hear details of the new setup, we have servers using vswitch too, and while I haven’t run into trouble myself I’d like to avoid it if possible. You know what they say about the wise learning from others’ experience!

permalink
report
reply
19 points

In the future please schedule your unplanned downtime around my (also unplanned) pooping schedule, thank you. Today’s experience was an almost barely noticeable inconvenience!

(Ofc I kid, lemm.ee is awesome!)

permalink
report
reply
14 points

Thanks for your hard work!

permalink
report
reply
13 points

Thanks for the update and your hard work!

permalink
report
reply

Meta (lemm.ee)

!meta@lemm.ee

Create post

lemm.ee Meta

This is a community for discussion about this particular Lemmy instance.

News and updates about lemm.ee will be posted here, so if that’s something that interests you, make sure to subscribe!


Rules:

  • Support requests belong in !support
  • Only posts about topics directly related to lemm.ee are allowed
  • If you don’t have anything constructive to add, then do not post/comment here. Low effort memes, trolling, etc is not allowed.
  • If you are from another instance, you may participate in discussions, but remain respectful. Realize that your comments will inevitably be associated with your instance by many lemm.ee users.

If you’re a Discord user, you can also join our Discord server: https://discord.gg/XM9nZwUn9K

Discord is only a back-up channel, !meta@lemm.ee will always be the main place for lemm.ee communications.


If you need help with anything, please post in !support instead.

Community stats

  • 879

    Monthly active users

  • 258

    Posts

  • 7.4K

    Comments

Community moderators