TL;DR possible thumbnail and local image outage tonight

Tonight I plan to update the image engine behind lemmy, called pictrs, to the latest stable version and to also perform a migration to a different database (sled to postgres).

In my testing there was minimal outage, but it did take quite some hours to do the migration (12+, though my test server is a lot slower than the production one).

Once the migration is complete, I intend to turn on a thumbnail cache cleaner that will significantly reduce the size of the thumbnail cache on the server. Lemmy currently saves many thumbnails locally, and never deletes them. This means we have over 260GB of images that get backed up each night for no reason. Lemmy makes it hard to tell the difference between cached images and user uploaded ones, but the cleaning tool will go through each post in the database, find the thumbnail associated with it, and delete it if it’s more than a month old.

Any questions, let me know!

Also, we are still having issues with lemmy.world. They are testing out new tools to block kbin traffic if they start to see huge amounts of activity coming from them, which will hopefully mitigate this to some extent. There is a tool running to help get posts and comments to us in real time, but it won’t work for votes which means you’ll see lots of lemmy.world posts and comments with zero votes, and the votes will be delayed in arriving by up to 1.5 days.

Feel free to ask questions about that too!

3 points

Migration has started!

permalink
report
reply
2 points

Morning update - migration is still running. I’m still not exactly sure what the impact is. Many thumbnails seem to come via other instances, and the lemmy.nz ones I can find seem to work but are probably cached by cloudflare. In any case, it should be minimal impact.

permalink
report
parent
reply
1 point

Afternoon update - migration is still running…

permalink
report
parent
reply
1 point

Night update - this is definitely taking longer than I expected…

permalink
report
parent
reply

Lemmy.nz Support

!support@lemmy.nz

Create post

Ask your questions here

Community stats

  • 30

    Monthly active users

  • 78

    Posts

  • 585

    Comments

Community moderators