Hacker News Reader

Comment by PaulKeeble | Hacker News Reader

The problem is the robots that do follow robots.txt its all the bots that don't. Robots.txt is largely irrelevant now they don't represent most of the traffic problem. They certainly don't represent the bots that are going to hammer your site without any regard, those bots don't follow robots.txt.

View on HN

That's what honeypots are for.

Deny /honeypot in your robots.txt

Add <a href="/honeypot" style="display:none" aria-hidden="true">ban me</a> to your index.html

If an IP accesses that path, ban it.

I like this. Adding now. Thanks!

> <a href="/honeypot" style="display:none" aria-hidden="true">ban me</a>

Unrelated meta question but is the aria tag necessarily since display: none; should be removing the content from the flow?

I would hope that all screen readers would respect display:none. The aria-hidden is for CYA, banning even one blind user would be quite bad optics (as is this sentence now that I think about it).

I wonder whether the path in the robots.txt (or maybe a <link> tag with a bogus rel attribute) would already be enough to make evil bots follow it. That would at least avoid accidental follows due to CSS/ARIA not properly working in weird constellations

There is an interesting article about AI tarpits that addresses a similar issue: https://arstechnica.com/tech-policy/2025/01/ai-haters-build-...

The argument is basically to have them scrape your website indefinitely wasting their resources for the bots that decide to ignore your robots.txt (or any bot if you desire)

While this may work today, the bots now more and more use if not the full headless rendering then at least apply CSS rules not to fetch invisible content.

Not sure why you were downvoted. I have zero confidence that OpenAI, Anthropic, and the rest respect robots.txt however much they insist they do. It's also clear that they're laundering their traffic through residential ISP IP addresses to make detection harder. There are plenty of third-parties advertising the service, and farming it out affords the AI companies some degree of plausible deniability.

Nobody has any confidence in ai to not ddos. That's why there have been dozens of posts about how bandwidth is becoming an issue for many websites as bots continuously crawl their sites for new info.

Wikipedia has backups for this reason. AI companies ignore the readily available backups and instead crawl every page hundreds of times a day.

I think debian also recently spoke up about it.

One way to test if the model respects robots.txt could be to ask the model if it can scrape a given URL. One thing that it doesn't address though is the scrapping of the training data for the model. That area feels more like a wild west.

Reply on news.ycombinator.com