Ask HN: Real-time backend bot handling? | Hacker News Reader

Ask HN: Real-time backend bot handling?

2 pointsshanebellone10 comments

Morning HN.

I'm looking to programmatically identify bots to serve 403s rather than content. For this project, an 80% reduction would be considered successful. Search bots will still retain access to content.

HTTP headers and real-time analytics will be combined to accomplish the task. I can measure the time between hits while considering the visitor's browser, operating system, and language specifications. I'm uncertain if there are legitimate use cases for atypical user agents?

How would you approach the task with backend technology (without JS, cookies, or external tools)?

View on HN

Bots that use headless Chrome will not likely be detected without special tools.

The poorly written bots can be detected looking at TCP header information such as MSS, window size, sequence numbers, TTL, packet length. [1] These are some of the things that CDN's like Cloudflare look at. This probably won't get you to your desired 80% but might get you half the way there.

Another method I have used to squash a handful of bots is to require HTTP/2.0. Many of the older bots only speak HTTP/1.1 or 1.0. The modern/common browsers can speak HTTP/2.0. This method may break some API CLI tools if your site has an API gateway. This can be worked around by putting the API gateway on its own load balancer or web servers. I should add that Google's crawler bots can not yet speak HTTP/2.0 so exclude them from this restriction.

Another method is to implement strict-SNI on your load balancers. Some bots can't deal with this yet and will end up on your default dummy site or just get an error depending on how one implements this.

[1] - https://github.com/p0f/p0f/

"Bots that use headless Chrome will not likely be detected without JS and other tools."

The bulk of bots seem to be running empty user agents which makes for a simple fix. I understand and agree that advanced bots (good or bad) will likely being impossible to detect. That's ok. I view it as "upping the ante".

"Another method I have used to squash a handful of bots is to require HTTP/2.0."

I'm at the quick, dirty, and effective stage so this is a solid recommendation. Thank you!

I'm at the quick, dirty, and effective stage so this is a solid recommendation. Thank you!

You're welcome. One caveat I should have mentioned is that this will break Google's crawlers as they for some reason can not yet speak HTTP/2.0. Bing will be fine. Archive.ph will be fine. One possible work-around would be to allow HTTP/1.1 for Google's CIDR blocks used by their crawlers not their cloud.

Some of Google's bots will identify themselves in the user-agent. Some will just look like an Android phone. They do this on purpose to see who is cloaking or changing content based on the visitor.

Another work-around for this is to have your static/indexed content on a site that permits HTTP/1.1 and then enforce HTTP/2.0 on the authenticated portion of your site. This division can be useful later on to optimize for caching static content.

I have the ability to make decisions before returning a response. The plan is to authorize desirable bots, default to 403 for unknown/unauthorized bots, and return content for the rest. This would allow me to serve Googlebot via 1.1 and default traffic to 2.0.

I'm thinking that a parent request's time-on-page (or time between requests) would indicate whether to serve a 403 or content during a subsequent hit. There are a few false positive scenarios to deal with they seem manageable.

Any thoughts on that particular approach?

Time-between-requests would seem reasonable except that bots will be crawling your site and others at the same time. This adds a delay combined with whatever crawl delay you specify in your robots.txt for search engine bots so I suspect you may get some false positives. Perhaps you could operate this in a log-only passive mode for a while to log what would have been blocked? The same applies to malicious bots. They can be crawling hundreds of sites at the same time. There is no way to guarantee your site gets the same threads and resources by the bots. Some of the malicious bots are using pools of LTE modems so even their IP's will be spread across a large CIDR block. That said, if you specify a crawl delay of 2 seconds in robots.txt and the bots do not obey this then perhaps it may be worth blocking them.

Allowing HTTP/1.1 for the bots that specify the google user-agent will not accommodate their bots that pretend to be Android phones and may result in Google flagging your site as potentially malicious. Some malicious bot tools will also use their user-agent in attempt to bypass authentication on some sites.

"Time-between-requests would seem reasonable except that bots ... I suspect you may get some false positives."

Solid point.

"Perhaps you could operate this in a log-only passive mode for a while to log what would have been blocked?"

I can and will. This seems to be the safest way forward.

"The same applies to malicious bots. They can be crawling hundreds of sites at the same time. There is no way to guarantee your site gets the same threads and resources by the bots. Some of them are using pools of LTE modems so even their IP's will be spread across a large CIDR block."

This is very true. Best case scenario: you block all malicious bots... but then they evolve. I'm viewing this from a resource efficiency perspective. I want to limit the cost of automated hits.

"That said, if you specify a crawl delay of 2 seconds in robots.txt and the bots do not obey this then perhaps it may be worth blocking them."

Which makes this a great idea. It's simple and dichotomous. All desirable bots will respect this declaration.

"Allowing HTTP/1.1 for the bots that specify the google user-agent will not accommodate their bots that pretend to be Android phones and may result in Google flagging your site as potentially malicious. Some malicious bot tools will also use their user-agent in attempt to bypass authentication on some sites."

Ah, I just realized Googlebot and these "android bots" use 1.1. Wouldn't be a problem otherwise. I'll leave this idea alone for now. Never anger the SERP gods. :D

Edit: It seems Googlebot does not accept the crawl-delay directive. Instead, it needs to be set in Search Console every 90 days.

https://support.google.com/webmasters/answer/48620?hl=en

> I'm uncertain if there are legitimate use cases for atypical user agents?

RSS readers and small search engines you haven't heard of but you're discriminating against :-)

> How would you approach the task with backend technology (without JS, cookies, or external tools)?

I solve this with client-side techniques (eg. recaptcha). Isolating and blocking IPs ranges from server logs (eg. drop anything from AWS & Azure subnets) might discourage bot operators.

"RSS readers and small search engines you haven't heard off but you're discriminating against :-)"

I'm 100% ok with that.

"I solve this with client-side techniques"

As a user, do you mind navigating these same approaches in the wild? Many time, I prefer leaving to solving a captcha.

> As a user, do you mind navigating these same approaches in the wild? Many time, I prefer leaving to solving a captcha.

Recaptcha doesn't show a challenge to the user. It just silently tells your backend what it thinks of the user.

Ah, my mistake. Thanks for the correction. :)

Reply on news.ycombinator.com