If you have known (and proven) legitimate traffic, give it a high QoS.
For never-been-seen IPs, provide a limited initial rate, with training based on experience.
Use tools such as the ASN Routeviews project to identify contiguous blocks of IP space (or any other source of BGP routing data, but the RV data are queryable via DNS and downloadable as zonefiles).
ID bad actors and either block or severely limit them.
Train up or down other traffic as appropriate.
This leaves you tracking large amounts of IP space. Doable in IPv4, somewhat more difficult under IPv6, though you're still probably going to be able to do something reasonable.
I'd like to see more of this pushed into the routing/networking layer, automatically, based on application-layer-based feedback. Maybe someday.