Hacker News Reader

Comment by ratmeadow | Hacker News Reader

Maybe a dumb question but what exactly is wrong with banning the IPs? Even if the bots get more IPs over time, surely storing a list of bans is cheaper than serving content? Is the worry that the bots will eventually cycle through so many IP ranges that you end up blocking legit users?

View on HN

It's often one IP (v4!) per one request. It's insane how many resources are being burned on this stupidity.

Part of the reason I did this is to get good numbers on how bad the problem is: A link maze is a great way to make otherwise very stealthy bots expose themselves.

Even if this is true how long can that be sustained before they start to be recycled? I bet the scrappers make a whole lot more requests than they have IPs

They are usually using residential IPs through SOCK5. I am not sure how they are getting these residential IPs but it is definitively suspicious.

So by blocking these IPs, you are blocking your users. (ie: in many coffeshops, I get the "IP Blocked" banner, my guess is that they are running software on unsuspecting users to route this traffic).

They use scammy providers like Bright Data[1] that let app authors embed their malware (for a compensation, I'm sure) which turns users' devices into crawler proxies.

[1]: https://brightdata.com/trustcenter/sourcing

> So by blocking these IPs, you are blocking your users.

There were 122 million residential internet connections in the US in 2024 so for an app with 1 million users the chance of affecting a single user is <1%.

[1] https://docs.fcc.gov/public/attachments/DOC-411463A1.pdf

Reply on news.ycombinator.com