Hacker News Reader

The presumption I’m already using cloudfare is a start. Is this a requirement for maintaining a simple website now?

Either that or Anubis (https://anubis.techaro.lol/docs), yes.

So these companies broke the internet

Which companies?

OpenAI, Anthropic, Google? No, their bots are pretty well behaved.

The smaller AI companies deploying bots that don't respect any reasonable rate limits and are scraping the same static pages thousands of times an hour? Yup

Anecdote, but at least for tiny little server hosting single public repository, none of these companies had 'well behaved' bots. It may be possible that they learned to behave better but I wouldn't know since my only possible recourse was to blacklist them all AND take the repo private.

Those are the small companies spoofing their user agent as the big companies to dodge countermeasures.

Reply on news.ycombinator.com