Hacker News Reader

Comment by autoexec | Hacker News Reader

As much as these companies should respect our preferences, it's very clear that they won't. It wouldn't matter to these companies if it was outright illegal, "pretty please" certainly isn't going to cut it. You can't stop scraping and the harder people try the worse their sites become for everyone else. Throwing up a robots.txt or llms.txt that calls out their bad behavior isn't a bad idea, but it's not likely to help anything either.

View on HN

In one of my robots.txt I have "Crawl-Delay: 20" for all User-Agents. Pretty much every search bot respect that Crawl-Delay, even the shaddy ones. But one of the most known AI bots launched a crawl requesting about 2 pages per second. It was so intense that it got banned by the "limit_req_" and "limit_rate_" of the nginx config. Now I have it configured to always get a 444 by user agent and ip range no matter how much they request.

> a crawl requesting about 2 pages per second. It was so intense [...]

Do 2 pages per second really count as "intense" activity? Even if I was hosting a website on a $5 VPS, I don't think I'd even notice anything short of 100 requests per second, in terms of resource usage.

I assumed that he meant per client. Having a limit of 2 pages a second for a single client seems like a reasonable amount to me.

If ypu open DevTools and visit any website these days, you'll be surprised.

In my scenario you request one single page from the proxy endpoint, and all other requests go straight to the static files and have no limits. I know than no human needs to request more than 1/s from the proxy, unless you are opening tabs frantically. So far, I only get praises about how responsive and quick the sites are: being harsh with the abusers means more resources for the regulars.

Downvoted for asking a completely reasonable question? Where am I?

Rookie question, how do you ban an ip range?

You can do it in a few places, but I use my network firewall for this I use PFSense at home, but there are many enterprise grade brands).

It's common to use the host's firewall as well (nftables, firewalld, or iptables).

You can do it at the webserver too, with access.conf in nginx. Apache uses mod_authz.

I usually do it at the network though so it uses the least amount of resources (no connection ever gets to the webserver). Though if you only have access to your webserver it's faster to ban it there than to send a request to the network team (depending on your org, some orgs might have this automated).

In your nginx, server section:

    deny 1.2.3.0/24;

And all 256 ips from 1.2.3.0 to 1.2.3.255 get banned. You can have multiple "deny" lines, or a file with "deny" and then include it.

It's better to do it at the firewall.

Reply on news.ycombinator.com