Generally speaking, in addition to the patterns you mentioned (invalid user-agent strings, rapid succession of same-IP access) there are also other patterns, such as failure to load referenced CSS or image files, or 100% failure on Captcha challenges.
"100% failure on Captcha challenges" -- are captchas generally deployed on click-through ads?
Very much so, after all your browser does without you having to tell it to do so. It all depends on whether the hassle is worth it for the bot writer.
Cloudflare has a mode (used in general for DDOS protection) which you can turn on and get stats on how many users pass a Captcha test.
Use http://phantomjs.org/ and the page will loaded by webkit.
The point was mostly that if it is not downloaded, then it's a bot (or any non-browser getter, at least), not that it's not a bot if it does.
Or maybe you were asking for a way to avoid bot detection ? :)