Metastable Failures in Distributed Systems

> Fast paths, caches, retries, failover, load balancing, and autoscaling all make the failure state less resource-efficient relative to the normal state, which makes the feedback loop worse. Beware of very high cache hit rates!

This scares me as someone who is working on performance tuning and is adding fast paths every other day....

The paper kinds of reminded me the "Gray failure" paper by the way.

https://blog.acolyer.org/2017/06/15/gray-failure-the-achille...

The HotOS'21 paper (Aleksey is one of the authors) is very much worth reading if you're building or maintaining large-scale distributed systems: https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s...

I wrote a blog post about this paper, too: https://brooker.co.za/blog/2021/05/24/metastable.html

One is reminded of Nassim Taleb's writing on efficiency and fragility. Long-lived systems (e.g. in biology) have a built in surplus, rather than operating everything at peak efficiency, because longevity is often determined not by how well you perform normally, but how well you perform during your worst crisis moments.

Here’s a simple one from my career. A calls B calls C. C makes a bad deploy. B passes through the errors from C and A retries against them. The retries push B’s utilization past the limit and instances start getting OOM killed. The C deploy is reverted. But the retries from A to B aren’t about C anymore, they’re about the fact that B is overloaded. A and B are in a neat little death spiral. We have to stop A, let B recover, and then slowly turn A back on.

After this incident we installed a per-instance rate limit on all services.

A very simple system is sufficient to demonstrate such metastable states. Consider a frontend supported by a database. Requests to the FE come in at a uniform rate below the ability of the database. Now there’s a peak in incoming requests. As parallel queries pile up, they start stepping on each other’s toes and the response time goes up. Given the temporarily stable incoming request rate, this means even more parallel queries and even slower response rate. Eventually the database response rate drops to the incoming request rate. If the peak of traffic subsides, the system sees a stable, usual, incoming request rate but a massively struggling database with a large number of parallel requests. This is the metastable state. When the db is restarted, purging the connections, everything returns to normal. Which can lead to the conclusion of the db platform being at fault. This is why you have queues or connection pools in front of your BE: to prevent dynamic complexity leaking over component boundaries and wreaking havoc all over the system.

Interesting article. It seems like if you had some kind of generic load shedding component that could monitor the difference between goodput and throughput and throttle load until the former increases it might help? Of course how to do this while guaranteeing it won't make the situation worse is another thing.

I had never seen the term “goodput” before. Kind of a clumsy portmanteau...

https://en.m.wikipedia.org/wiki/Goodput