4 points

TBF, pushing a site to the public while adding a “no scrapping” rule is a bit of a shitty practice; and pushing it and adding a “no scrapping, unless you are Google” is a giant shitty practice.

Rules for politely scrapping the site are fine. But then, there will be always people that disobey those, so you must also actively enforce those rules too. So I’m not sure robots.txt is really useful at all.

permalink
report
reply
21 points
*

How would a site make itself acessible to the internet in general while also not allowing itself to be scraped using technology?

robots.txt does rely on being respected, just like no tresspassing signs. The lack of enforcement is the problem, and keeping robots.txt to track the permissions would make it effective again.

I am agreeing, just with a slightky different take.

permalink
report
parent
reply
1 point

User agent catching is rather effective. You can serve different responses based on UA.

So generally people will use a robots.txt to catch the bots that play nice and then use useragents to manage abusers.

permalink
report
parent
reply
68 points

No it’s not, what a weird take. If I publish my art online for enthusiasts to see it’s not automatically licensed to everyone to distribute. If I specifically want to forbid entities I have huge ethical issues with (such as Google, OpenAI et. al.) from scraping and transforming my work, I should be able to.

permalink
report
parent
reply
-28 points

Nothing in my post (or in robots.txt) has any relation to distributing your content.

permalink
report
parent
reply
23 points
*

What else would they scrape your data for? Sure some could be for personal use but most of the time it will be to redistribute in a new medium. Like a recipe app importing recipes.

permalink
report
parent
reply
5 points

People really should be providing a sitemap.xml file

permalink
report
parent
reply
5 points

She should get a library card.

permalink
report
reply
2 points

She tried but she couldn’t read the application.

permalink
report
parent
reply
59 points

TikTok spider has been a real offender for me. For one site I host it burred through 3TB of data over 2 months requesting the same 500 images over and over. It was ignoring the robots.txt too, I ended up having to block their user agent.

permalink
report
reply
25 points

Are you sure the caching headers your server is sending for those images are correct? If your server is telling the client to not cache the images, it’ll hit the URL again every time.

If the image at a particular URL will never change (for example, if your build system inserts a hash into the file name), you can use a far-future expires header to tell clients to cache it indefinitely (e.g. expires max in Nginx).

permalink
report
parent
reply
7 points

Thanks for the suggestion, turns out there are no cache headers on these images. They indeed never change, I’ll try that update. Thanks again

permalink
report
parent
reply
56 points

Perfect use of the meme

permalink
report
reply
27 points

time to fill sites code with randomly generated garbage text that humans will not see but crawlers will gobble up?

permalink
report
reply
13 points

I don’t think it’s a bad idea but it’s largely dependent on the crawler. I can’t speak for AI based crawlers, but typical scraping targets specific elements on a page or grabbing the whole page and parsing it for what you’re looking for. In both instances, your content is already scrapped and added to the pile. Overall, I have to wonder how long “poisoning the water well” is going to work. You can take me with a grain of salt, though; I work on detecting bots for a living.

permalink
report
parent
reply
21 points

I work on detecting bots for a living.

You should just tell people you’re a blade runner.

permalink
report
parent
reply
14 points

I’m a blade runner. 😁

permalink
report
parent
reply
3 points
*

Until you realize that you are paying for access fees/network

permalink
report
parent
reply
2 points

Accessibility tho

permalink
report
parent
reply

Lemmy Shitpost

!lemmyshitpost@lemmy.world

Create post

Welcome to Lemmy Shitpost. Here you can shitpost to your hearts content.

Anything and everything goes. Memes, Jokes, Vents and Banter. Though we still have to comply with lemmy.world instance rules. So behave!


Rules:

1. Be Respectful

Refrain from using harmful language pertaining to a protected characteristic: e.g. race, gender, sexuality, disability or religion.

Refrain from being argumentative when responding or commenting to posts/replies. Personal attacks are not welcome here.


2. No Illegal Content

Content that violates the law. Any post/comment found to be in breach of common law will be removed and given to the authorities if required.

That means:

-No promoting violence/threats against any individuals

-No CSA content or Revenge Porn

-No sharing private/personal information (Doxxing)


3. No Spam

Posting the same post, no matter the intent is against the rules.

-If you have posted content, please refrain from re-posting said content within this community.

-Do not spam posts with intent to harass, annoy, bully, advertise, scam or harm this community.

-No posting Scams/Advertisements/Phishing Links/IP Grabbers

-No Bots, Bots will be banned from the community.


4. No Porn/Explicit

Content


-Do not post explicit content. Lemmy.World is not the instance for NSFW content.

-Do not post Gore or Shock Content.


5. No Enciting Harassment,

Brigading, Doxxing or Witch Hunts


-Do not Brigade other Communities

-No calls to action against other communities/users within Lemmy or outside of Lemmy.

-No Witch Hunts against users/communities.

-No content that harasses members within or outside of the community.


6. NSFW should be behind NSFW tags.

-Content that is NSFW should be behind NSFW tags.

-Content that might be distressing should be kept behind NSFW tags.

If you see content that is a breach of the rules, please flag and report the comment and a moderator will take action where they can.


Also check out:

Partnered Communities:

1.Memes

2.Lemmy Review

3.Mildly Infuriating

4.Lemmy Be Wholesome

5.No Stupid Questions

6.You Should Know

7.Comedy Heaven

8.Credible Defense

9.Ten Forward

10.LinuxMemes (Linux themed memes)


Reach out to

All communities included on the sidebar are to be made in compliance with the instance rules. Striker

Community stats

  • 14K

    Monthly active users

  • 11K

    Posts

  • 244K

    Comments