Comment by iso@lemmy.dbzer0.com on Where does Lemmy store upvotes, downvotes, and replies?

Does ActivityPub send those to other instances, or does ActivityPub only send the original post and the rest (upvotes, downvotes, replies) are stored only on the original server where the post was made?

haven’t worked with AP yet, but as a webdev I’m certain it’s original server only. Syncing upvotes between nodes would be an insane datavolume and one hell to properly keep in sync to begin with.

They are synced. There is an insane data volume, yes. It is hell.

no way, that’s a massive oof o.O

Yeah. A lot of hand-wringing has gone on about it, e.g. https://gist.github.com/jdarcy/60107fe4e653819138396257df302eef. I’ll post this and then show you a video of server activity that results.

Here is a screencast of what happens to my 2 core server when I post something - https://kglitch.social/activitypub_cpu_and_net.mp4.

I run a single user instance, more or less, so there is little chance of some other user causing this load.

Some of it will be due to the way Kbin is built but I believe any software using ActivityPub to communicate will run into similar issues sooner or later, especially with network traffic usage.

Damn that’s crazy. Thanks for the demo

What the fuckkk haha this is crazy. Hold on. I’m testing it on my instance now, let’s see if Lemmy acts differently

Completly off-topic, but what’s that dope af htop replacement?

My instance has 800 users, is 4 months old, and the database only is over 30GB. It is an insane amount of data.

How much RAM does your server have to handle a 30 GB database?

I’m a bad example. I haven’t properly tuned the settings, currently RAM will grow to whatever is available.

I’m very lucky, the instance is running in a proxmox container alongside some other fediverse servers (run by others), on dedicated hardware in a datacentre. The sysadmin has basically thrown me plenty of spare resources since the other containers aren’t using them and RAM not used is wasted, so I’ve got 32GB allocated currently. I still need to restart once a week or that RAM gets used up and the database container crashes.

It’s been on my list of things to do for a while, try some different postgres configs, but I just haven’t got around to it.

I know a couple of months back lemmy.world were restarting every 30 mins so they didn’t use up all the RAM and crash. I presume some time and some lemmy updates later that’s no longer the case.

I know some smaller servers get away with 2gb of RAM, and we should be able to use a lot less than 32GB if I actually try to tune the postgres config.

There is a postgres command to show the size of each table. Most likely it is from activity tables which can be cleared out to save space.

After the second-to-last update the database shrunk and I was under the impression there was some automatic removal happening. Was this not the case?

It’s helpful info for others but personally I’m not that worried about the database size. The size of the pictrs cache is much more of a concern, and as I understand it there isn’t an easy way to identify and remove cache images without accidentally taking out user image uploads.

Yes there is automatic removal so if you have enough disk space, no need to worry about it.

The pictrs storage only consists of uploads from local users, and thumbnails for both local and remote posts. Thumbnails for remote posts could theoretically be wiped and loaded from the other instance, but they shouldnt take much space anyway.

Yes there is automatic removal so if you have enough disk space, no need to worry about it.

What triggers this? My DB was about 30GB, then the update shrunk it down to 5GB, then it grew back to 30GB.

The pictrs storage only consists of uploads from local users, and thumbnails for both local and remote posts. Thumbnails for remote posts could theoretically be wiped and loaded from the other instance, but they shouldnt take much space anyway.

I’d be pretty confident that the 140GB of pictrs cache I have is mostly cache. There are occasionaly users uploading images, but we don’t have that many active users, I’d be surprised if there was more than a few GB of image uploads in total out of that 140GB. We just aren’t that big of a server.

The pictrs volume also grows consistently at a little under 1GB per day. I just went and had a look, in the files directory there are 6 directories from today (the day only has a couple of hours left), and these sum to almost 700MB of images and almost 6000 files, or a little over 100KB each.

The instance has had just 27 active users today (though of course users not posting will still generate thumbnails).

While the cached images may be small, it adds up really quick.

As far as I can tell there is no cache pruning, as the cache goes up pretty consistently each day.

The activities table is cleared out automatically every week, items older than 3 months are deleted. During the update only a smaller number of rows was migrated so the db temporarily was slower. You can manually clear older items in sent_activity and received_activity to free more space.

Actually Im wrong about images, turns out that all remote images are mirrored locally in order to generate thumbnails. 0.19 will have an option to disable that. This could use more improvements, the whole image handling is rather confusing now.

[This comment has been deleted by an automated system]

Thanks, that’s very informative. How does this work since ActivityPub can be used for other things, e.g., Mastodon? They ignore any “Type” entries that they don’t support?

[This comment has been deleted by an automated system]

It does sync them, I can even query all of your votes on my local DB for every community my instance is tracking.

A loosely moderated place to ask open-ended questions

Search asklemmy 🔍

If your post meets the following criteria, it’s welcome here!

Open-ended question
Not offensive: at this point, we do not have the bandwidth to moderate overtly political discussions. Assume best intent and be excellent to each other.
Not regarding using or support for Lemmy: context, see the list of support communities and tools for finding communities below
Not ad nauseam inducing: please make sure it is a question that would be new to most members
An actual topic of discussion

Looking for support?

Looking for a community?

Lemmyverse: community search
sub.rehab: maps old subreddits to fediverse options, marks official as such
!lemmy411@lemmy.ca: a community for finding communities

_Icon _by _{@Double_A@discuss.tchncs.de}

Where does Lemmy store upvotes, downvotes, and replies?

Asklemmy

!asklemmy@lemmy.ml

Community stats

Community moderators