Is there a reason you cannot just outsource this problem to AWstats/Webalizer/Google Analytics/etc? It's a hard problem, and those packages have done quite a bit of work to solve it.

(From experience, switching from - IIRC - AWstats to Webalizer reduces your traffic by a lot, as the latter's robot detection algorithm catches more robots. IIRC - it's possible that it's the other way round...)`