Hacker News Reader

Comment by otherme123 | Hacker News Reader

In one of my robots.txt I have "Crawl-Delay: 20" for all User-Agents. Pretty much every search bot respect that Crawl-Delay, even the shaddy ones. But one of the most known AI bots launched a crawl requesting about 2 pages per second. It was so intense that it got banned by the "limit_req_" and "limit_rate_" of the nginx config. Now I have it configured to always get a 444 by user agent and ip range no matter how much they request.

View on HN

Rookie question, how do you ban an ip range?

You can do it in a few places, but I use my network firewall for this I use PFSense at home, but there are many enterprise grade brands).

It's common to use the host's firewall as well (nftables, firewalld, or iptables).

You can do it at the webserver too, with access.conf in nginx. Apache uses mod_authz.

I usually do it at the network though so it uses the least amount of resources (no connection ever gets to the webserver). Though if you only have access to your webserver it's faster to ban it there than to send a request to the network team (depending on your org, some orgs might have this automated).

In your nginx, server section:

    deny 1.2.3.0/24;

And all 256 ips from 1.2.3.0 to 1.2.3.255 get banned. You can have multiple "deny" lines, or a file with "deny" and then include it.

It's better to do it at the firewall.

> a crawl requesting about 2 pages per second. It was so intense [...]

Do 2 pages per second really count as "intense" activity? Even if I was hosting a website on a $5 VPS, I don't think I'd even notice anything short of 100 requests per second, in terms of resource usage.

I assumed that he meant per client. Having a limit of 2 pages a second for a single client seems like a reasonable amount to me.

If ypu open DevTools and visit any website these days, you'll be surprised.

In my scenario you request one single page from the proxy endpoint, and all other requests go straight to the static files and have no limits. I know than no human needs to request more than 1/s from the proxy, unless you are opening tabs frantically. So far, I only get praises about how responsive and quick the sites are: being harsh with the abusers means more resources for the regulars.

Downvoted for asking a completely reasonable question? Where am I?

Reply on news.ycombinator.com