Exactly what I think is happening here, confusion between web crawler bots and the other sort of bots.