What does your stack look like? If you're running a ruby app, check out rack-attack: https://github.com/kickstarter/rack-attack

It was created by kickstarter to fend off these kind of attacks.