Right now, robots.txt on lemmy.ca is configured this way
User-Agent: *
Disallow: /login
Disallow: /login_reset
Disallow: /settings
Disallow: /create_community
Disallow: /create_post
Disallow: /create_private_message
Disallow: /inbox
Disallow: /setup
Disallow: /admin
Disallow: /password_change
Disallow: /search/
Disallow: /modlog
Would it be a good idea privacy-wise to deny GPTBot from scrapping content from the server?
User-agent: GPTBot
Disallow: /
Thanks!
Yes, please.
We can’t stop LLM developers from scraping our conversations if they’re determined to do so, but we can at least make our wishes clear. If they respect our wishes, then great. If they don’t, then they’ll be unable to plead ignorance, and our signpost in the road (along with those from other instances) might influence legislation as it’s drafted in the coming years.
I’m on board for this, but I feel obliged to point out that it’s basically symbolic and won’t mean anything. Since all the data is federated out, they have a plethora of places to harvest it from - or more likely just run their own activitypub harvester.
I’ve thrown a block into nginx so I don’t need to muck with robots.txt inside the lemmy-ui container.
# curl -H 'User-agent: GPTBot' https://lemmy.ca/ -i
HTTP/2 403
Yes. Ban them.
if ($http_user_agent = "GPTBot") {
return 403;
}
I would have thought so too, but == failed the syntax check
2023/08/07 15:36:59 [emerg] 2315181#2315181: unexpected "==" in condition in /etc/nginx/sites-enabled/lemmy.ca.conf:50
You actually want ~ though because GPTBot is just in the user agent, it’s not the full string.
Strangely, = works the same as == with nginx. It’s a very strange config format…
https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#if
1000% yes. Please block them.
Is this even possible without all federated instances also prohibiting them?
