We also had some actual attacks so we made a system that detect anomalies (like more than 50rps per IP) and raises alert.
...which was thwarted by frontend developers again, as they loaded few hundred tiny icons at once that triggered that alert routinely, and only thru http2 multiplexing their idiotic design patents haven't bitten them before.
Performance metrics were taking a dive and triggered automated alerts - average response time jumped from 200ms to 2000, 8000, and eventually began approaching 15000, at which point requests were timing out all over the place. At first I was wondering if my recently-deployed MR was responsible, but upon further investigation, it was from one developer on another team doing two very dumb things with their application prototype:
- Constantly retrying the same query with no filtering and setting the maximum allowed page size
- Sending massive quantities of small queries for specific individuals (50-100 per second) which quickly hit the rate limit, and immediately resuming after the rate limit expired
Some developer outreach was required...