Hacker News Reader

Comment by Animats | Hacker News Reader

I've put fair queuing into an web API. Requests queue by IP address; if you have a request in the mill, any further requests are processed behind those from other IP addresses. This handles single-source overloads very well. It doesn't require tuning; there are no fixed time constants or rate limits. For about a month, someone was pounding on the system with thousands of requests per minute. This caused no problems to other users. I didn't notice for several days.

One would think that this would be a standard feature in web servers.

View on HN

This is an interesting approach to request rate limiting - the simplicity and lack of tuning is definitely appealing.

Did you do anything to mitigate the scenario where multiple users are behind the same IP address? With this approach I would worry about locking out all users in a NAT when a single user misbehaves.

It's also scary to me to process something "last" - under high enough load you might never get back to the (potentially briefly) abusive user? Did you attempt to guarantee some minimum passthrough rate even for misbehaving users?

As with fair queuing in routers [1], you have to avoid infinite overtaking. If someone has have a request in progress, their other requests are stuck behind that one until the request in progress is serviced. Then your next request is eligible to be processed.

[1] https://tools.ietf.org/html/rfc970

This is a great idea. And unlike rate limits per user, you should get better utilization of the web server. A drawback is request latencies and queue sizes could get out of control without a feedback mechanism to the client. I would probably want some active queue management to return a 429 if something stays in the queue too long. You could also put a single rate limiter in front of the web server, to prevent server overload.

I return HTTP error 429 (Too Many Requests) when a single IP address has 100 requests queued. Anybody who gets into that situation is using the API incorrectly.

Where in your application stack did you implement this?

What were the hard kinks you had to figure out when coding this system?

I'm a fellow developer, would be interested in hearing you describe your system in more detail.

The queue is in a MySQL in-memory table, of all things. The requests are slow (they take tens of seconds, because they go out and scan a web site), so this is feasible. For smaller requests, you'd need a more direct mechanism.

So you have to hash by IP on your load balancer? You can't do a round Robin hash. That seems to be the downside i can think of.

Reply on news.ycombinator.com