Hacker News Reader

Comment by dawnerd | Hacker News Reader

Looking at my logs for all of my sites and this isn’t a global truth. I see multiple ai crawlers hammering away requesting the same pages many, many times. Perplexity and Facebook are basically nonstop.

View on HN

I just looked at the logs for a site, and I saw PerplexityBot is looking at the robots.txt and ignoring it. They don't provide a list of IPs to verify if it is actually them. Anyway, just for anyone with PerplexityBot in their user agent, they can get increasingly bad responses until the abuse stops.

Perplexity is exceptionally bad because they say they respect the robots.txt but clearly don't. When pressed on it they basically shrug and say too bad not put stuff in public if you don't want it crawled. They got a UA block in cloudflare and seems like that did the trick.

Interesting. Now they seem to claim that not only they follow robots.txt for crawling, but that they also broke under pressure and made the unfortunate decisions to have user requests follow robots.txt too.

https://www.perplexity.ai/de/hub/technical-faq/how-does-perp...

User Agent block just means they'd spoof their user agent.

That generally gives you even more trouble with cloudflare. Behaving in any way inconsistent with your UA string is one of the easiest methods of identifying bots.

Yeah you can use headless browsers, but then you're also using orders of magnitude more compute, and that's not really something that scales.

The best way to avoid ending up in captcha-land is to say who you are, and respect robots.txt.

What do you mean by many, many times?

Reply on news.ycombinator.com