My cheap and dirty way of dealing with bots like that is to block any IP address that accesses any URLs in robots.txt. It's not a perfect strategy but it gives me pretty good results given the simplicity to implement.
You need to impose cost. Set up QoS buckets, slow suspect connections down dramatically (almost to the point of timeout).