EDIT

TO EVERYONE ASKING TO OPEN AN ISSUE ON GITHUB, IT HAS BEEN OPEN SINCE JULY 6: https://github.com/LemmyNet/lemmy/issues/3504

June 24 - https://github.com/LemmyNet/lemmy/issues/3236

TO EVERYONE SAYING THAT THIS IS NOT A CONCERN: Everybody has different laws in their countries (in other words, not everyone is American), and whether or not an admin is liable for such content residing in their servers without their knowledge, don’t you think it’s still an issue anyway? Are you not bothered by the fact that somebody could be sharing illegal images from your server without you ever knowing? Is that okay with you? OR are you only saying this because you’re NOT an admin? Different admins have already responded in the comments and have suggested ways to solve the problem because they are genuinely concerned about this problem as much as I am. Thank you to all the hard working admins. I appreciate and love you all.


ORIGINAL POST

You can upload images to a Lemmy instance without anyone knowing that the image is there if the admins are not regularly checking their pictrs database.

To do this, you create a post on any Lemmy instance, upload an image, and never click the “Create” button. The post is never created but the image is uploaded. Because the post isn’t created, nobody knows that the image is uploaded.

You can also go to any post, upload a picture in the comment, copy the URL and never post the comment. You can also upload an image as your avatar or banner and just close the tab. The image will still reside in the server.

You can (possibly) do the same with community icons and banners.

Why does this matter?

Because anyone can upload illegal images without the admin knowing and the admin will be liable for it. With everything that has been going on lately, I wanted to remind all of you about this. Don’t think that disabling cache is enough. Bad actors can secretly stash illegal images on your Lemmy instance if you aren’t checking!

These bad actors can then share these links around and you would never know! They can report it to the FBI and if you haven’t taken it down (because you did not know) for a certain period, say goodbye to your instance and see you in court.

Only your backend admins who have access to the database (or object storage or whatever) can check this, meaning non-backend admins and moderators WILL NOT BE ABLE TO MONITOR THESE, and regular users WILL NOT BE ABLE TO REPORT THESE.

Aren’t these images deleted if they aren’t used for the post/comment/banner/avatar/icon?

NOPE! The image actually stays uploaded! Lemmy doesn’t check if the images are used! Try it out yourself. Just make sure to copy the link by copying the link text or copying it by clicking the image then “copy image link”.

How come this hasn’t been addressed before?

I don’t know. I am fairly certain that this has been brought up before. Nobody paid attention but I’m bringing it up again after all the shit that happened in the past week. I can’t even find it on the GitHub issue tracker.

I’m an instance administrator, what the fuck do I do?

Check your pictrs images (good luck) or nuke it. Disable pictrs, restrict sign ups, or watch your database like a hawk. You can also delete your instance.

Good luck.

122 points

Perhaps someone should create a script to purge orphan images

permalink
report
reply
46 points

Seems like the logical fix

permalink
report
parent
reply
57 points

The logical fix would be to delete them automatically when unused for longer than let’s say 24 hours. That should be in the lemmy code, and we should not depend on 3rd party utilities to do that.

permalink
report
parent
reply
11 points

You can submit a patch upstream

permalink
report
parent
reply
2 points

Well, yeah. We’re saying the same thing. A script to fix this running outside Lemmy is a quick fix. But this same process should be built into Lemmy itself.

permalink
report
parent
reply
12 points
*

Or, just tighten up the api such that uploaded pictures have a relatively short TTL unless they become attached to a post or otherwise linked somewhere.

A script is a fine stopgap measure, but we should try to treat the cause wherever possible, instead of simply addressing the symptom.

permalink
report
parent
reply
6 points
*

What’s the practical difference? In both cases you’re culling images based on whether they’re orphaned or not.

If you’re suggesting that the implementation be based on setting individual timers instead of simply validating the whole database at regular intervals, consider whether or not the complexity of such a system is actually worth the tradeoff.

“Complexity comshmexity”, you might say. “Surely it’s not a big deal!”. Well… what about an image that used to belong to a valid post that later got deleted? Guess you have to take that edge case into account and add a deletion trigger there as well! But what if there were other comments/posts on the same instance hotlinking the same image? Guess you have to scan the whole DB every time before running the deletion trigger to be safe! Wait… wasn’t the whole purpose of setting this up with individual jobs to avoid doing a scripted DB scan?

permalink
report
parent
reply
3 points

There are mechanisms that exist in a LOT of services for handling TTL expiry and any relevant purging that needs to be done.

That said, a cursory look at the pict-rs project doesn’t appear to have any provision for TTL, so it’s probably going to have to be done as a cron job anyways - or at least triggered by the lemmy service when an image upload isn’t used in an instance-local lemmy post within some reasonable interval.

Note that I’m specifically including “in an an instance-local post” because I am assuming admins don’t want to provide free cloud image hosting to random internet people for arbitrary non-lemmy use.

permalink
report
parent
reply
8 points

Very much needed.

permalink
report
parent
reply
104 points
*

FYI to all admins: with the next release of pict-rs, it should be much easier to detect orphaned images, as the pict-rs database will be moved to postgresql. I am planning to build a hashtable of “in-use” images by iterating through all posts and comments by lemm.ee users (+ avatars and banners of course), and then I will iterate through all images in the pict-rs database, and if they are not in the “in-use” hash table, I will purge them.

Of course, Lemmy can be improved to handle this case better as well!

permalink
report
reply
43 points

Did you opened an issue on github?

You are wording this as a clickbait news article.

You find an issue, you report it to the right channel, you notify it. Good. This is how software development work, with active community reporting issues.

But why using such tone?

permalink
report
reply
33 points

I’m not on GitHub. Nor is a lot here. I’m wording it this way so the issue gets the attention it deserves. Anyway, everybody already knows about this but nobody understood the consequences. Same reason why there’s no option to disable image caching. These issues should have been addressed the moment image uploading was made available in Lemmy. It was just overlooked because of how tiny the platform was then.

It’s funny because last month Mastodon CSAM was a hot topic in the Fediverse and people were being defensive about it. Look where we are now. Has Mastodon addressed the CSAM issue? Did they follow the recommendations made by that paper? I don’t think so. There wouldn’t be an open GitHub issue about it. Will Lemmy be like Mastodon or will it addressed the concerns of its users?

permalink
report
parent
reply
23 points
*

Creating a product of any size is about planning.

If you notify here, your information will be lost in 2 days. People forget, and move on to the next hot topic. Relevant stakeholders might very well completely miss this post, because they are not 24/7 on lemmy.

The way to make it more relevant is going in the place where the planning is done, i.e. Github for lemmy. Open an issue there, explain the problem and describe possible solution. Come back to lemmy, link the issue and ask people to react to it (i.e. show it is relevant for them).

This is the best way to obtain what you ask. Social media platforms are too broad and fuzzy for tracking real issues.

This is also why you see a lot of work is done on performances of sql of lemmy backend, because most issues in the past on github concerned that.

This is my suggestion. If you really care about this being implemented, open a ticket on github and follow the discussion there. If you see there is not enough traction ask help to fellow lemmings.

Suggestions for the github issue are:

  • be very specific
  • be polite
  • suggest solutions

If your solution is good, great, if not, people are more willing to think about a problem to show stranger on the internet they are wrong

permalink
report
parent
reply
6 points

Feel free to open the issue on my behalf. I am not a software developer. You seem to know more about this. I’m just reminding people something that I and many others have observed months ago.

permalink
report
parent
reply
4 points

so the issue gets the attention it deserves.

The other best way to do this is to actually submit the issue in the appropriate location so the Lemmy devs can track and respond to it.

It’s been 7 hours, it can’t be that hard to make a github account and format this post into an actually helpful github issue.

permalink
report
parent
reply
3 points

Because there’s already an issue dated July 6: https://github.com/LemmyNet/lemmy/issues/3504

Like I said, people already know about this months ago.

permalink
report
parent
reply
2 points
*

Are you aware of the consequences of your actions? You didn’t inform the people who can fix this issue of the potential impact, no. You informed the Lemmy community that they can upload whatever they want, and some of them are pedophiles. Not cool at all. Responsible disclosure ain’t a thing outside of cybersecurity I suppose, though irresponsible disclosure is prevalent everywhere. Very irresponsible.

permalink
report
parent
reply
2 points

Rogues are very keen in their profession, and know already much more than we can teach them.

permalink
report
parent
reply
0 points

How would they address your concerns? The chances that one of the devs follows you is nonexistent, I would wager. Instead of using the proper channels to inform them, you did the exact opposite and posted it someplace they are almost guaranteed not to see it.

permalink
report
parent
reply
2 points

It’s on the GitHub issue tracker already. Did you not read the post?

permalink
report
parent
reply
43 points
*
Deleted by creator
permalink
report
reply
18 points

Good point but also consider disabling pictrs until they fix the caching problem!

permalink
report
parent
reply
5 points
*
Deleted by creator
permalink
report
parent
reply
5 points

Is there a description on how to disable pictrs?

permalink
report
parent
reply
4 points

Remove it from docker compose.

permalink
report
parent
reply
38 points

I’m an instance administrator, what the fuck do I do?

There’s one more option. The awesome @db0@lemmy.dbzer0.com has made this tool to detect and automatically remove CSAM content from a pict-rs object storage.

https://github.com/db0/lemmy-safety

permalink
report
reply
24 points

This is a nice tool but orphaned images still need to be purged. Mentioned on the other thread that bad actors can upload spam to fill up object storage space.

permalink
report
parent
reply
7 points

That is also very true. I think better tooling for that might come with the next pict-rs version, which will move the storage to a database (right now it’s in an internal ky-value storage). Hopefully that will make it easier to identify orphaned images.

permalink
report
parent
reply
5 points

I tried getting this to run in a container, but I was unable to access my GPU in the container. Does anyone have any tips on doing that?

permalink
report
parent
reply
1 point

Sorry I haven’t ran this myself yet nor have any experience with that kind of issues. But may I ask why you were concerned with running it inside of a container? Seems rather unnecessary to me.

permalink
report
parent
reply
6 points

Running anything in a container isn’t necessary. It just makes it easier to run, as it comes with all the dependencies. And if you decide you don’t want it anymore, you can just remove the container and it and all its dependencies are gone, which is really clean. It also makes the environment extremely repeatable, so people on all distros can run it with the exact same steps. And you don’t need to worry about what version of python you have and if it’s compatible with the dependencies. For example, the dependencies for this script require python 3.10 exactly. You can’t use 3.9 or 3.11.

So really the only reason was I wanted to make it easier for everyone.

permalink
report
parent
reply
1 point

Not sure how you’re trying to run it in a container, but the answer would depend on a bunch of different factors. Nvidia has a utility you can install that assists in exposing the GPU to the container, documentation found here.

If you’re using docker compose to run it as a service, there’s a doc page for that too. Note that it uses the previous page I mentioned as prerequisite.

There’s another way to get it working from within kubernetes that comes up every now and then on stackoverflow.

If it’s Intel or AMD, no idea if this still applies.

permalink
report
parent
reply
1 point

Yes, this is exactly what I had trouble with. The Nvidia container runtime seems to not support my distro. But even when I tried running it on my Ubuntu machine, I was getting tons of dead links using Nvidia’s instructions. And even when I fixed the links. I was getting issues like the apt repository was throwing errors. IIRC, it was some kind of signature issue, and I’m not sure I want to ignore that, especially considering I had to fudge the URL.

I’m thinking the best option is to build from source, but I don’t think that’s easier than just running this in a non-container.

permalink
report
parent
reply
3 points

You need a GPU for that. Most $5 VPSs don’t have that.

permalink
report
parent
reply
4 points

Yeah I know. It’s supposed to be ran from your computer, not the VPS.

permalink
report
parent
reply
1 point
*

Would I mount the the pictrs folders as a network folder locally?

permalink
report
parent
reply

Lemmy

!lemmy@lemmy.ml

Create post

Everything about Lemmy; bugs, gripes, praises, and advocacy.

For discussion about the lemmy.ml instance, go to !meta@lemmy.ml.

Community stats

  • 315

    Monthly active users

  • 1.1K

    Posts

  • 14K

    Comments

Community moderators