In the past months, there’s a been a issue in various instances where accounts would start uploading blatant CSAM to popular communities. First of all this traumatizes anyone who gets to see it before the admins get to it, including the admins who have to review to take it down. Second of all, even if the content is a link to an external site, lemmy sill caches the thumbnail and stores it in the local pict-rs, causing headaches for the admins who have to somehow clear that out. Finally, both image posts and problematic thumbnails are federated to other lemmy instances, and then likewise stored in their pict-rs, causing such content to be stored in their image storage.

This has caused multiple instances to take radical measures, from defederating liberaly, to stopping image uploads to even shutting down.

Today I’m happy to announce that I’ve spend multiple days developing a tool you can plug into your instance to stop this at the source: pictrs-safety

Using a new feature from pictr-rs 0.4.3 we can now cause pictrs to call an arbitary endpoint to validate the content of an image before uploading it. pictrs-safety builds that endpoint which uses an asynchronous approach to validate such images.

I had already developed fedi-safety which could be used to regularly go through your image storage and delete all potential CSAM. I have now extended fedi-safety to plug into pict-rs safety and scan images sent by pict-rs.

The end effect is that any images uploaded or federated into your instance will be scanned in advance and if fedi-safety thinks they’re potential CSAM, they will not be uploaded to your image storage at all!

This covers three important vectors for abuse:

  • Malicious users cannot upload CSAM to for trolling communities. Even novel GenerativeAI CSAM.
  • Users cannot upload CSAM images and never submit a post or comment (making them invisible to admins). The images will be automatically rejected during upload
  • Deferated images and thumbnails of CSAM will be rejected by your pict-rs.

Now, that said, this tool is AI-driven and thus, not perfect. There will be false positives, especially around lewd images and images which contain children or child-topics (even if not lewd). This is the bargain we have to take to prevent the bigger problem above.

By my napkin calculations, false positive rates are below 1%, but certainly someone’s innocent meme will eventually be affected. If this happen, I request to just move on as currently we don’t have a way to whitelist specific images. Don’t try to resize or modify the images to pass the filter. It won’t help you.

For lemmy admins:

  • pictrs-safety contains a docker-compose sample you can add to your lemmy’s docker-compose. You will need to your put the .env in the same folder, or adjust the provided variables. (All kudos to @Penguincoder@beehaw.org for the docker support).
  • You need to adjust your pict-rs ENVIRONMENT as well. Check the readme.
  • fedi-safety must run on a system with GPU. The reason for this is that lemmy provides just a 10-seconds grace period for each upload before it times out the upload regardless of the results. A CPU scan will not be fast enough. However my architecture allows the fedi-safety to run on a different place than pictrs-safety. I am currently running it from my desktop. In fact, if you have a lot of images to scan, you can connect multiple scanning workers to pictrs-safety!
  • For those who don’t have access to a GPU, I am working on a NSFW-scanner which will use the AI-Horde directly instead and won’t require using fedi-safety at all. Stay tuned.

For other fediverse software admins

fedi-safety can already be used to scan your image storage for CSAM, so you can also protect yourself and your users, even on mastodon or firefish or whatever.

I will try to provide real-time scanning in the future for each software as well and PRs are welcome.

Divisions by zero

This tool is already active now on divisions by zero. It’s usage should be transparent to you, but do let me know if you notice anything wrong.

Support

If you appreciate the priority work that I’ve put in this tool, please consider supporting this and future development work on liberapay:

https://liberapay.com/db0/

All my work is and will always be FOSS and available for all who need it most.

-28 points

Sounds like progress, but please consider using a term other than “whitelist” when describing a list of allowed values. While the use of blacklist predates references to black as a race, allowlist is a reasonable alternative that doesn’t reinforce viewing black as less than or unwanted and white as allowed.

permalink
report
reply
13 points

Allowlist and Blocklist are also more intuitive to people who haven’t heard the terms before.

permalink
report
parent
reply
6 points
*

I’ve honestly always found “allowlist” and “blocklist” to feel like forced compound words, and I don’t see why “list” is necessary at all. For example, just saying “allowed” and “blocked” both implies it’s a list and is more intuitive than any of the *list terms.

Personally I have no stake in the battle, but I do wish people would use the most intuitive terms for the situation at least (whatever they are, for example “enabled”/“disabled” or “included”/“excluded”) instead of blanket ctrl+f on everything.

permalink
report
parent
reply
3 points

That’s a good point, and I hadn’t thought about that angle, that there just isn’t a reason for the terms to exist in the first place.

“In the red” and “in the black” is another pair that isn’t intuitive to me at all and I have to look up every time.

permalink
report
parent
reply
3 points

Wahhhhhhh

permalink
report
parent
reply
26 points

Making things that were never about race into things about race, just to have one more reason to be potentially offended by, is not productive and doesn’t help anyone.

By exercising enough mental gymnastics almost any term could be twisted into something supposedly offensive. The real solution to that problem: don’t do mental gymnastics.

permalink
report
parent
reply
11 points

Oh the fuck with this nonsense!

permalink
report
parent
reply
16 points
*

Oh come on… The origin of blacklist was centuries before “black” became the term for a person of colour. And on a thread about CSAM…

permalink
report
parent
reply
10 points

Do you ever get tired of twisting yourself into a pretzel every time you want to be offended?

permalink
report
parent
reply
18 points

Those are technical terms that have nothing to do with race or even humans.

permalink
report
parent
reply
8 points

Holy fuck

permalink
report
parent
reply
30 points
*
Deleted by creator
permalink
report
reply
14 points
*

For abuse detection, you need to use a service that has been vetted by an actual lawyer.

Lmao because every tech company out there is toooootallly doing that lmao. Come onnnnn give me a break, you’re on the programming.dev instance and you’re saying this shit? Have you ever shipped anything ever? You’re calling for something above and beyond anything that actually happens in reality.

The weirdest part of what you’re demanding is that your demand makes fediverse LESS protected. Imperfectly implemented protection is better than NO protection and yet you think people are safer legally by implementing nothing at all? You’re out of your fucking mind.

permalink
report
parent
reply
10 points
Deleted by creator
permalink
report
parent
reply
15 points
*

Drop the ad hominem attacks and argue the point

Just say “insult” ffs. Speak like a normal human being. This isn’t a debate and you win absolutely nothing by using debatebro reddit language.

Nobody has to argue with your stupid ass making absurd demands on an account with 10 whole comments in its entire history. It’s incredibly obvious wrecker behaviour.

permalink
report
parent
reply
-2 points
*
Deleted by creator
permalink
report
parent
reply
4 points
Deleted by creator
permalink
report
parent
reply
30 points

Are you a lawyer? This feels like FUD.

I strongly advise anyone against using this software in production, as you will be on the hook for anything this software doesn’t catch.

So if you don’t use this software, you’re not on the hook for the pictures that this tool doesn’t catch?

permalink
report
parent
reply
16 points
Deleted by creator
permalink
report
parent
reply
23 points

I think you’re confused by the purpose of that statement. When the authors say not to use it for anything important, they’re basically trying to waive liability (informally). It’s kind of like how every open source license has a statement like:

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED …

If you use an open source project for air traffic control software, and a bug causes a bunch of people to die, that’s your fault, not the author of the software. The CLIP people are essentially saying that you shouldn’t use their software to build something that requires a lot of accuracy since it probably wasn’t designed to be as accurate as you need it to be.

But what I’m wondering is why you’re being so dramatic about this. You’re claiming that it’s highly dangerous/reckless/risky to use it, yet hand waving over the why.

permalink
report
parent
reply
-9 points
*

On Reddit there were always so many concern Karen’s peddling FUD when you say anything critical of pedos. For the longest time on Reddit all you had to do was see if the user also posted in pedofriends.

It was so obvious.

Any attempt to do anything that might impact a pedophile and there was quickly a brigade saying why that was a wrong.

No. Any action that supports pedophiles is morally wrong and we should NOT let perfect be the enemy of good when trying to make it more difficult to be a pedophile or support pedophiles.

If you have a solution that’s better, implement it and share it as Open Source.

Otherwise crawl back in your pedo hole at reddit and leave us alone.

permalink
report
parent
reply
8 points
Deleted by creator
permalink
report
parent
reply
-6 points
*

I called concern troll and then mentioned that on Reddit the subreddit pedofriends uses concern trolling as a tactic to prevent people from doing anything about pedophiles.

I never said you were a pedophile, but way to out yourself as overly concerned with people being labeled pedos cuz you have such a big heart for protecting pedophiles.

permalink
report
parent
reply
4 points

While the OP’s concerns were largely… dubious(?), simply going through with literally whatever solution is just as big of a problem. I haven’t reviewed anything about this implementation in depth, thus can’t make any statements towards quality or truthfulness (except that 1% FP seems very optimistic), but you shouldn’t simply assume it will solve your problem without introducing another. Having said that, the author appears very well intentioned and has experience on the subject, so the veredict shouldn’t take long.

permalink
report
parent
reply
4 points

The solution is to try.

In dev it’s called “fail fast”

You try several solutions to the problem and iterate until you find the best working option then optimize.

There’s no harm in an independent open source dev trying to solve a problem. Even if they fail there might be a kernel of useful code in a novel way.

You absolutely shouldn’t just settle on one option without exploring several.

But being stuck in a loop of over planning, waiting for legal, or outsourcing the problem is why everyone hates waterfall and just has small teams try with “agile” now.

1% failure is just the start. You should never call a beta shit cuz it fails sometimes. Failure is part of learning and improving.

permalink
report
parent
reply
14 points

For abuse detection, you need to use a service that has been vetted by an actual lawyer.

Name one. That exists and already works on Fediverse instances.

permalink
report
parent
reply
-1 points
*
Deleted by creator
permalink
report
parent
reply

🤡

permalink
report
parent
reply
10 points
*

That’s not how this works.

So you can’t name one either.

If you’re running an instance, it’s your responsibility to find an appropriate tool.

That’s exactly what the admin here did, and what’s more, he did so because he was forced to do so by a lack of “appropriate” or even adequate tools.

Hence the straightforward question you failed to answer.

I don’t run an instance. CSAM is but one of the many reasons why. But I have been paying attention to the discussions regarding the flood of it here, and the impossibilities involved in starting from scratch with preventing/blocking this on a federated instance.

But for reasons I cannot begin to fathom, and with an intense interest in seeing this anti-CSAM tool remain unused, you are blithely sailing by all that with a demand for using a tool you personally could not even name and obviously does not exist in acceptable form, or it would already have been gladly implemented.

Glad he’s ignoring you and carrying on. I think I’ll do the same.

permalink
report
parent
reply
5 points
4 points

It’s free.

But only for Cloudflare customers

permalink
report
parent
reply
1 point
*
Deleted by creator
permalink
report
parent
reply
0 points

Fantastic work!

My first thought tho is of revenge. Is there any way to have it automatically report the incident to the FBI? Address of the uploader, etc

permalink
report
reply
36 points
*

Not unless lemmy and pict-rs devs provide a lot more functionality. However be aware that most hits will be false positives. You would be sending a lot of garbage over to the FBI unless you review every hit manually.

permalink
report
parent
reply
11 points

What about false positives?

permalink
report
parent
reply
1 point

Well someone would have to actually look at it if they want to prosecute

permalink
report
parent
reply
1 point
Deleted by creator
permalink
report
parent
reply
13 points

What if the troll is uploading from outside the USA ? I don’t think spaming authorities with reports will help…

permalink
report
parent
reply
2 points

You’re probably right

permalink
report
parent
reply
5 points

Also, if it reports a false positive, you’re sending an innocent person’s IP and other info to the authorities. I imagine the user won’t appreciate that.

permalink
report
parent
reply
-5 points
*

AI based with high false positive rate. Fantastic. These tools are great but impacting regular use of the platform is going to drive people away. Lemmy is at a critical state of needing to onboard users to plateau or grow, I feel this will not be good despite the good intentions.

It’s not like you can just re-post what gets taken down. The magic box’s response is final

permalink
report
reply
9 points

I suppose on boarding users trumps dealing with CSAM. Thanks for clearing that up.

permalink
report
parent
reply
2 points

Literally unhinged take lmfao

Reasses your fucking priorities dude. Fucking child abuse or users on the website. Imo you deserve a global ban if you’re a serious on this take and not just being a 4head

permalink
report
parent
reply
-3 points

Someone disagrees with me on the internet! Global ban!

permalink
report
parent
reply
0 points

Nah just people who are cool with sacrificing the removal of fucking child porn for user numbers.

Pedo.

permalink
report
parent
reply
7 points

permalink
report
reply

/0

!div0@lemmy.dbzer0.com

Create post

Meta community. Discuss about this lemmy instance or lemmy in general.

Service Uptime view

Community stats

  • 186

    Monthly active users

  • 172

    Posts

  • 1.9K

    Comments

Community moderators