Hacker News Reader

Comment by nneonneo | Hacker News Reader

I wonder if one solution here could be to turn up Anubis difficulty if the first page hit is not one of the obvious entry-points to cgit. It could even have a little hint that says something to the effect of “go visit the home page if this is taking too long”. (Better not to ban them entirely, in case people really did click on some random link e.g. in a news story or mailing list message).

Distributed scrapers are going to generally try and hit their assigned list of pages; it’s a bigger waste of time if they have to go off to visit other pages first to get the cookie challenge.

View on HN

> I wonder if one solution here could be to turn up Anubis difficulty if the first page hit is not one of the obvious entry-points to cgit.

Anubis has a fairly capable "policy" system. you can place something like this in your policy.json:

``` { "bots": [ { "action": "WEIGH", "expression": "path.startsWith(\"/expensive/endpoint\")", "name": "scrutinize-expensive-endpoints", "weight": { "adjust": 20 } } ] } ```

another thing smaller sites benefit from -- where the load induced by crawlers tends to be bursty (e.g. as they discover new expensive endpoints to crawl) -- is to adjust the difficulty up/down to maintain a steady system load.

``` { "bots": [ { "action": "WEIGH", "expression": "load_15m <= 16.0", "name": "sustained-low-load", "weight": { "adjust": -10 } }, { "action": "WEIGH", "expression": "load_5m >= 24.0", "name": "intermittent-high-load", "weight": { "adjust": 10 } }, ] } ```

Almost all of my visits to cgit instances are through direct deep links. Hard to imagine someone randomly browsing git listings

The kernel folks likely have a good profile on what page people trigger Anubis on (i.e. what page people hit first). From that they could make heuristics about what pages are likely to be useful deep links.

Keep in mind that Anubis will rarely inconvenience an actual user who visits the site often; it’s meant to keep out first-time scrapers trying to grab a few pages from their queue.

Reply on news.ycombinator.com