Hacker News Reader

Comment by oefrha | Hacker News Reader

I tried with a nonexistent path, here are the user agents I got:

  Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36

It's apparently built to evade detection.

View on HN

the more reasonable assumption is that it uses a third party crawl service.

but still, it doesn't matter, it's acting on behalf of a user, and you aren't entitled to know what software your users run. One consequence of putting stuff on the public internet is that it's like public.

what sort of evasion technique is being used here, random user agents on each attempt?

> what sort of evasion technique is being used here

Using a user-agent that looks like a desktop user-agent, rather than including the name of the actual project/product that is being used.

I understand why though, plenty of websites block anything that looks like non-desktop/mobile user-agents, so makes sense. Besides the pragmatic reason, I also agree with menacingly that it doesn't matter, people should be able to use whatever user-agent they want to access the content you've put publicly on the internet.

Reply on news.ycombinator.com