Hacker News Reader

Make a new certificate, let crawlers blast you and add those IPs to a block list.

these old network security techniques don't really work anymore. the common bots are at known IP ranges, the problem bots are all on datacenter + residential proxies.

Why would blocking those be a problem?

there are 150M+ of them and you'll be taking out a lot of human users with it

modern blocking is behaviour / heuristic based

There are 150 million bots all using residential IP addresses?

because you are blocking all of Comcast, Verizon, T-Mobile, British Telecom, ....

at the end you have blocked every network with human visitors and only datacenter IPs can access your site.

The proxies rotate IP every day, so you either have ineffective blocking or you block the whole network.

My site is not for americans so I don't care about blocking american isps

You think they only use American networks?

In my experience, these aren't the crawlers from legit companies, so they have infinite IPs via residential botnets/proxies.

edit: 'nikcub beat me to it by 30 seconds :)

Reply on news.ycombinator.com