A live zoo of bots visiting one site, with a trap for those ignoring robots.txt

This is a fun breakdown of traffic and behaviors :)

One potential improvement: using reverse DNS to check the Agent user-agent-string. User Agent strings are unauthenticated, so can't be attributed to a specific crawler without a check.

I am working on a similar breakdown of traffic understanding, and have found a small but interesting amount of 'forged' agent strings. Mostly from the bots doing 404 knocks for pages that don't exist (vulnerability scanning behavior).