Hacker News Reader

Comment by scrollaway | Hacker News Reader

By nature it won't filter out bots, however.

One of my clients' website traffic is composed of over 75% bot traffic. Server-side logs are unusable for anything other than site performance.

View on HN

> One of my clients' website traffic is composed of over 75% bot traffic. Server-side logs are unusable for anything other than site performance.

I'm unclear how broad you intend that second sentence to be, but there's still a ton of info you can glean just from server-side logs:

- Referrer info, and, by extension, popular search terms being used to find your site;

- Paths on your site causing 5xx errors (so pages which might be triggering an error in a server-side script)

- Paths on your site causing 4xx errors, and associated referrers (might be broken links on your own site; might be stale search engine indexing)

- Mobile vs desktop access statistics

Finding this data among a bunch of bot-induced noise might be annoying, but if they're good bots and sending proper UA headers specifying their botness, it's easy enough to filter out. Even otherwise there might be typical bot-like behavior you can find and account for such as not sending a referrer header or trying to access known exploitable PHP scripts (in which case you should block that IP address for a few hours or days - there are programs which can do this sort of thing automatically but frustratingly I can't recall the name of one off the top of my head right now).

Granted, a lot of this can be spoofed, but I'm pretty sure the number of people sending spoofed referral or UA headers is dwarfed by the number of those (like me) who block Google Analytics and similar cruft entirely.

No, you're right, I shouldn't have written something so dismissive. (I do include error tracing as part of "performance" for what it's worth but those have their own system from within the app itself)

Frankly I would love to see some serious low-config solutions to analyzing server-side logs. Oh, especially Fastly. Client in question uses Fastly and it blew my mind to find out that there was nothing in place to answer simple questions such as "what are the slowest paths to respond", "which paths are a cache hit most often", "which paths are most hit overall", etc. And being able to look at various dimensions such as browser, bot traffic, country of origin, etc. If you have suggestions…

Any log analyzer will tell you which path is the most hit. For slowest paths, I think a server daemon could theoretically log how long it took to serve the page from request in to last byte out, but I don't know if any of them do that - you might have to set up a custom format for logging, and then from there you'd need to tell your analyzer how to interpret that field. For cache hits, I guess it'd depend on what sort of cache you have in mind, but that might be something you could only effectively log at the application level.

| frustratingly I can't recall the name of one off the top of my head right now

fail2ban

> By nature it won't filter out bots, however.

I have two questions about this:

- Since you know they are bots, why couldn't you filter them?

- On the other hand, couldn't there be bots that run JavaScript which would be tracked client-side?

> Since you know they are bots, why couldn't you filter them?

A decent amount can be filtered by UA. This is inconvenient because UA is a very large piece to log and index on, so you need to do UA processing to do anything useful with it and … well by that point it just becomes a chore and I suspect there's good logging services that do this better than you'd spend your time doing yourself.

> On the other hand, couldn't there be bots that run JavaScript which would be tracked client-side?

They exist but they're more rare by nature, because running JS at bot scale is expensive.

> well by that point it just becomes a chore and I suspect there's good logging services that do this better than you'd spend your time doing yourself.

Which is why the OP suggested using tools like goaccess.

Goaccess does have a separate panel for Crawlers/Bots(1) but I think it's based on the UA.

There is also an option to ignore IPs/Referrers too which can work very well for such cases.

(1) https://goaccess.io/features#:~:text=Spot%20aggressive%20hos...

The middle way is using something like Pirsch's API [0] from your backend.

[0] https://pirsch.io/

Can I evade tracking by disgusing my traffic as bot traffic? Sounds like a startup idea.

You're likely going to end up implementing a poor man's Tor.

Oh I suspect you'll have a decent amount of sites not serving you ads or analytics if you put "googlebot" or "chrome lighthouse" in your UA.

If you use googlebot you’ll likely just get blocked altogether.

Using Lighthouse might be an interesting experiment though.

Reply on news.ycombinator.com