How do you tell if it's a bot? Just by user-agent string or repeated slamming by IP address, or is there some other technique?
A combination of things. Off the top of my head:
* IP address ranges. Are they all from, e.g., EC2 instances?
* Are images and other content (e.g. CSS) linked from the page loaded? How long does the client take to do so? Are other pages being viewed in line with what you've seen from other users?
* What combinations of user agents are you seeing?
* What's in the headers being sent by the browser? Any foreign language support, or oddities when it comes to not accepting compressed content? Is the referer header always the same? Does it always vary?
* Do you see "normal" user behaviour that indicates activity such as clicking the back button, then clicking the ad again a few seconds later once their brain has kicked in? (nb: not sure if this is possible for Facebook with the way their site works).
* Are there any patterns in the page access times? Is there anything to indicate a cron job is kicking off once an hour at the same time?
Generally speaking, in addition to the patterns you mentioned (invalid user-agent strings, rapid succession of same-IP access) there are also other patterns, such as failure to load referenced CSS or image files, or 100% failure on Captcha challenges.
"100% failure on Captcha challenges" -- are captchas generally deployed on click-through ads?
Very much so, after all your browser does without you having to tell it to do so. It all depends on whether the hassle is worth it for the bot writer.
Use http://phantomjs.org/ and the page will loaded by webkit.
Cloudflare has a mode (used in general for DDOS protection) which you can turn on and get stats on how many users pass a Captcha test.
The point was mostly that if it is not downloaded, then it's a bot (or any non-browser getter, at least), not that it's not a bot if it does.
Or maybe you were asking for a way to avoid bot detection ? :)
Although they stated they built their own analytics software, so there's also a chance that it wasn't working as they thought it was.
Later:
"So we did what any good developers would do. We built a page logger. Any time a page was loaded, we'd keep track of it. You know what we found? The 80% of clicks we were paying for were from bots. That's correct. Bots were loading pages and driving up our advertising costs"
I think they use a different definition of bot. Otherwise they wouldn't have to use a second logger ...