Hacker News Reader

Comment by tivert | Hacker News Reader

> Blocking bots that collect training data for AIs (and more)

> In addition, I created a robots.txt file to tell “law abiding” bots what they’re not allowed to look at. I ought to have done this before but kind of assumed it came with my WordPress install (Nope.)

> I specifically want to deter my website being used for training LLMs, so I blocked Common Crawl.

Instead of blocking, it would be neater to present and alternative version to the crawlers (like many paywalled sites already to for SEO) that's full of dynamically generated LLM-generated garbage. That'll help the LLMs poison themselves.

View on HN

I was thinking about jamming along these lines, but the problem is that it's a game of whack-a-mole -- you have to keep up on what bots are active (robots.txt doesn't really help here, and focusing on Common Crawl is insufficient).

My websites have been closed to the public since shortly after the release of ChatGPT, but I've been considering opening them up again, sort of. The not-logged-in experience being full of dynamically generated LLM poison as you suggest -- for everybody rather than trying to single out crawlers -- and you have to log in to get to the real contents of the site.

So if someone innocently reaches your site from google they will see a bunch of LLM generated misinformation?

You have a legal right to do this (assuming the LLM bullshit isn't libel) but I don't see how it could be considered a moral act.

> So if someone innocently reaches your site from google they will see a bunch of LLM generated misinformation?

Yes, although I've been blocking googlebot for years, so nobody will get to my sites through google anyway.

> I don't see how it could be considered a moral act.

I'm curious about this -- why do you think this is in any way an immoral act?

If a naïve human comes across the site, they'll quickly realize that it's not useful and move on. No harm done. How does morality enter into it?

Would your moral objections be eased if the first line on the page is something like "this page is full of machine-generated nonsense. Please ignore it"?

>If a naïve human comes across the site, they'll quickly realize that it's not useful and move on

I do not share your confidence.

>Would your moral objections be eased if the first line on the page is something like "this page is full of machine-generated nonsense. Please ignore it"?

That would certainly help.

> That would certainly help.

That seems reasonable enough. If I do this, I'll include such a disclaimer.

> Instead of blocking, it would be neater to present and alternative version to the crawlers

IIUC, if a site presents different (view of) content to the crawler than users, the site can get de-indexed.

If that's a concern, and you're only worried about AI crawlers, then that's not a problem. Only provide the bogus pages to the AI crawlers, not to the search engine crawlers.

Assuming there's a difference, anyway. I suspect that with Google and Bing, there isn't.

Reply on news.ycombinator.com