Fail2ban doesn't help when the attacker/abuser rotates their IP address constantly. I now look for aberrations in a few http headers they often neglect to spoof in their attempt to act like an honest human.
Be very careful that fail2ban isn't actually exhausting your own resources faster than you believe you're exhausting the attackers' resources...
The attacker even went to the trouble of spoofing some custom headers in our API client.
Our eventual solution was to a) attack our own auth credentials first, identify any users with leaked creds (from other services) and force a password reset for them. b) disallow users from setting common leaked passwords. c) make the auth checking request as low-cost as possible, and scalable separately from the main application. d) when an attack request is detected, bypass the relatively-expensive real cred check but return the same failure response (including timing) as a real failure. e) build a secondary requirement in the auth flow that can be transparently enabled when under high volume attack.
This works, so far. It sheds the volume to the application, and has low-to-zero impact on legit users. This took a couple weeks away from feature development though!
How many active users (to an order of magnitude; no need for precise numbers of course) does this service have? 100k IPs sounds pretty costly to burn, so I'm curious how important one needs to be before that's considered worth it.
And could you say what type of IP addresses those were? Did it look residential such as from compromises computers (botnet), do they rent lots of IP addresses temporarily from aws/netcup/alibaba, or is it a mix such that neither category has the overwhelming majority?
If it's all server IP ranges for a service where end users normally log in, you could apply entirely different rate limits to those than to residential IPs, for example. Hence I'm wondering how these cred stuffing attacks are set up
And of course a ton of open proxies.