Block? Nope, robots.txt does not block the bots. It’s just a text file that says: “Hey robot X, please do not crawl my website. Thanks :>”
I disallow a page in my robots.txt and ip-ban everyone who goes there. Thats pretty effective.
humans typically don’t visit [website]/fdfjsidfjsidojfi43j435345 when there’s no button that links to it
Imagine posting a rule that says “do not walk on the grass” among other rules and then banning anyone who steps on the grass with the thought process that if they didn’t obey that rule they were likely disobeying other rules. Except the grass is somewhere that no one would see unless they actually read the rules. The rules were the only place that mentioned that grass.
Is the page linked in the site anywhere, or just mentioned in the robots.txt file?
Not sure if that is effective at all. Why would a crawler check the robots.txt if it’s programmed to ignore it anyways?
Unfortunate indeed.
“Can AI bots ignore my robots.txt file? Well-established companies such as Google and OpenAI typically adhere to robots.txt protocols. But some poorly designed AI bots will ignore your robots.txt.”
typically adhere. but they don’t have to follow it.
poorly designed AI bots
Is it a poor design if its explicitly a design choice to ignore it entirely to scrape as much data as possible? Id argue its more AI bots designed to scrape everything regardless of robots.txt. That’s the intention. Asshole design vs poor design.
This is why I block in a htaccess:
# Bot Agent Block Rule
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (BOTNAME|BOTNAME2|BOTNAME3) [NC]
RewriteRule (.*) - [F,L]
This is still relying on the bot being nice enough to tell you that it’s a bot; it could just not.
Exactly. The only truly effectively way I’ve ever found to block bots is to use a service like Akamai. They have an add-on called Bot Manager that identifies requests as bots in real time. They have a library of over 1000 known bots and can also identify unknown bots built on different frameworks, bots that impersonate well known bots like Googlebot, etc. This service is expensive, but effective…
This article lies to the reader, so it earns a -1 from me.
Lies, as in that it’s not really “blocking” but a mere unenforceable request? If you meant something else could you please point it out?
That is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered.
This does not block anything at all.
It’s a 1994 “standard” that requires voluntary compliance and the user-agent is a string set by the operator of the tool used to access your site.
https://en.m.wikipedia.org/wiki/Robots.txt
https://en.m.wikipedia.org/wiki/User-Agent_header
In other words, the bot operator can ignore your robots.txt file and if you check your webserver logs, they can set their user-agent to whatever they like, so you cannot tell if they are ignoring you.
robots.txt does not work. I don’t think it ever has - it’s an honour system with no penalty for ignoring it.
I have a few low traffic sites hosted at home, and when a crawler takes an interest they can totally flood my connection. I’m using cloudflare and being incredibly aggressive with my filtering but so many bots are ignoring robots.txt as well as lying about who they are with humanesque UAs that it’s having a real impact on my ability to provide the sites for humans.
Over the past year it’s got around ten times worse. I woke up this morning to find my connection at a crawl and on checking the logs, AmazonBot has been hitting one site 12000 times an hour, and that’s one of the more well-behaved bots. But there’s thousands and thousands of them.
Cloudflare just announced an AI Bot prevention system: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/
