Currently, what I do is that when an IP requests insane amounts of URLs on my server (especially when its all broken urls causing 404s) I look up the IP and then block the whole organization.

For example today some bot from the range 14.224.0.0-14.255.255.255 got crazy and caused a storm of 404s. Dozens per second for hours on end. So I blocked the range like this:

iptables -A INPUT -m iprange --src-range 14.224.0.0-14.255.255.255 -j DROP

That's probably not the best way and might block significant parts of whole countries. But at least it keeps my service alive for now.

What do others here do to protect their servers?