79

How to block AI Crawler Bots using robots.txt file(www.cyberciti.biz)

posted
by
Avatar
Cynicus Rex@lemmy.ml
in
Avatar
privacy@lemmy.ml
64 comments
hidereport
Sort:
HotTopControversialNewOld
You are viewing a single thread.
View all comments View context
Avatar
Cynicus Rex@lemmy.mlOP
-2 points
*

Lies, as in that it’s not really “blocking” but a mere unenforceable request? If you meant something else could you please point it out?

permalink
report
parent
reply
Avatar
Da Bald Eagul@feddit.nl
37 points

That is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered.

permalink
report
parent
reply
Avatar
JackbyDev@programming.dev
0 points

Is it a lie or a simplification for beginners?

permalink
report
parent
reply
Avatar
mox@lemmy.sdf.org
4 points

Assuring someone that they have control of something and the safety that comes with it, when in fact they do not, is well outside the realm of a simplification. It’s just plain false. It can even be dangerous.

permalink
report
parent
reply
Avatar
thanks_shakey_snake@lemmy.ca
13 points

Lie. Or at best, dangerously wrong. Like saying “Crosswalks make cars incapable of harming pedestrians who stay within them.”

permalink
report
parent
reply
Show more comments
Avatar
Eager Eagle@lemmy.world
1 point

the word disallow is right there

permalink
report
parent
reply

Privacy

!privacy@lemmy.ml

Create post

A place to discuss privacy and freedom in the digital world.

Privacy has become a very important issue in modern society, with companies and governments constantly abusing their power, more and more people are waking up to the importance of digital privacy.

In this community everyone is welcome to post links and discuss topics related to privacy.

Some Rules

  • Posting a link to a website containing tracking isn’t great, if contents of the website are behind a paywall maybe copy them into the post
  • Don’t promote proprietary software
  • Try to keep things on topic
  • If you have a question, please try searching for previous discussions, maybe it has already been answered
  • Reposts are fine, but should have at least a couple of weeks in between so that the post can reach a new audience
  • Be nice :)

Related communities

Chat rooms

much thanks to @gary_host_laptop for the logo design :)

Community stats

  • 3.9K

    Monthly active users

  • 3K

    Posts

  • 78K

    Comments

Community moderators