Architecture Patterns: The Circuit-Breaker | Hacker News Reader

Architecture Patterns: The Circuit-Breaker

lab.scub.net

94 pointskiyanwang16 comments

I came to think of circuit breakers as a crude tool that has a very limited set of uses, much smaller than what it's used for. It might work as a last resort, but it should never be deployed as your first way to deal with failures.

If at all possible, APIs should communicate server health status back to their clients, and clients should have ways to deal with partial outages of supporting services and systems.

Also, that sequence diagram should have been a state diagram. Don't people teach proper UML in schools these days?

I had the same problem understanding the state diagram but didn’t even realize he used the wrong type of UML diagram :))

UML is taught, sometimes, but once you get into the industry it is a coin flip as to whether or not it is properly used - if it is used at all.

I would think the first resort in most of the non-flooding-related cases[1] is backpressure. If you've got too many requests coming in, be sure that internally you start parking the excess incoming requests in some low-resource-usage configuration (e.g., simply noting the incoming TCP connection but then not even read()ing from it until you are ready), and let the system push back naturally by the way its latency increases. This obviously naturally pairs with both client-side and server-side timeouts, giving both a vote in how long they're willing to wait.

This is generally what you want. Actively breaking the connections if I think I'm overloaded is only something I'd add if there is some additional constraint on the system that suggests it's a good idea due to some specific local concerns. Circuit breaking produces a much sharper (for lack of a better term) performance graph, where instead of degrading gracefully like backpressure will, you suddenly start getting active failures.

I've built a lot of backpressure in my systems; I'm yet to need a circuit breaker. YMMV, of course, but I see some people referring to it as a basic tool and I would disagree; the basic tool is backpressure.

[1]: Flooding is its own thing. In general, to weather DOS attacks but still have the service functional requires another system or likely set of systems in front of the service that can handle the DOS issues. I say this just to point out that circuit breakers aren't really for this case, because no system can on its own handle a full-on DOS flood, no matter what code you write, when the DOS flood may literally be coming in at a speed where it is faster than you can even reject connections, let alone do any work.

Yeah, I don't think circuit breakers are really the appropriate choice in most of the situations the article is describing. Rate limiting and backpressure seem like better options most of the time.

The way I see it, circuit breakers are safety devices. They're for when you need to keep a system in a safe control region and are wiling to sacrifice some reliability in order to achieve that. e.g. preventing customers from accidentally turning your globally distributed whatever into a DDOS platform or limiting the blast radius when infrastructure automation decides it should delete everything.

Oh yes! I used response times as a proxy for backend health a couple times. Still, it's better if the service tells you there is some degradation in a way the client can understand and handle. Sometimes, only some of the requests need to be slowed down or paused, leading to some limited functionality but not a complete loss of service.

Yeah, they kind of are really liked ... by the service maintainers who don't have to do the front end work to tell the bad news.

The article seems to gloss over that part, which is understandable, because it's probably targeted at... service coders.

So how does this work on the front end? One way or another you're communicating to the user that the company/service isn't up to their expectations, SLA be damned.

It also gets into the classic issue of queuing requests and dedicating resources. Imagine this: you have 100 customers, but 10% of them take 5-10x longer to serve than the others. This is pretty common. Like airline checkin or border crossings, some people will take 2 minutes, some 20 minutes.

But... if you don't have any way of keeping a service handler dedicated to fast queries, all the slow bombs will saturate your workers. So even though one or tow of five service workers could service 80% of the work load quickly, if you can't identify the fast requests from the slow ones and keep a fast track for the fast ones, you're service descends to a response time of handling the worse cases (assuming they are frequent just... enough).

Circuit breaking doesn't really help that.

When people think of circuit breakers, I think most people think of the automated things like hystrix / istio circuit breaking.

I feel like those have their place, but in practice are often overkill and they are complex enough that it's hard to keep them loaded in my brain when I rarely think about them. What I find to be more broadly useful is the ability to change things like timeout and retry on the fly. Or other more manual kill switches via dynamic configuration.

When something is on fire and my service is getting overloaded by another service that is crushing it with retries, I can then make a surgical change. Not automatic, but still a good MTTR. I think this is one of the better ways to use what we're building at Prefab.

Agreed. Configurability (which is a proxy of circuit-breaking), especially at the border, is often overlooked when we build systems. If it's not configurable it probably (but not always) have some "smarts" to deal with problems.

I’m starting to think circuit breakers are not long for this world.

You get your front end reliably serving web pages to thousands of people and the bevy of search engine bots out there, and you start wondering why only the front end has to be robust like this. Why don’t your services use the same set of tools to approach a similar level of reliability?

In particular, if you have seven services or apps all calling the same service, a pretty common arrangement is one circuit per client-service pair, so you have five different clusters trying to track and react to the health of a sixth, when a load balancer in front of that service could be handling it instead.

They aren’t robustness in depth. They’re a coping mechanism for not having it.

circuit breakers /can/ make sense, if done rationally. kinda like that line in fallbacks vs failovers where "if the second system is so reliable why bother with the first?"

https://aws.amazon.com/builders-library/avoiding-fallback-in...

(fallback: e.g., using direct DB instead of cache, failover: second cache/API)

> if the second system is so reliable why bother with the first.

Hmm. That's a difficult one to unpack.

Sometimes services need to be reliable in different dimensions. And that first service just might not be able to scale high enough to do some more esoteric operation on the data. Though I could see someone arguing that from the user's perspective the first and second system are trying to pretend to be the same thing.

I think the services I have the hardest time shooting are the ones that exist to prevent priority inversion. I maintain a batch processing system I have a hard time justifying in some ways. I get more value out of it as a guineapig for new tools and techniques we may want to apply more broadly, than I get from the actual task it performs, but I certainly don't want the core system trying to handle it either. It'd be a lot more noise. So I want to kill it, and I want to keep it.

Some tasks have escalating priority over time, and trying to mix them with tasks that don't can be problematic.

I’ve also seen people reinventing the wheel (poorly), seemingly to avoid existing and proven technology, like writing a bespoke DB load balancer / health checker in Node using Redis and Dynamo, instead of just standing up HAProxy.

“My JavaScript will surely be more performant and less buggy than production-grade C that’s existed for decades.”

I guess it’s more ‘I don’t need another layer in my stack which I don’t know how to configure; I just need this one feature, how hard can it be’.

Nginx is not a joy to configure but it’s probably easier than hystrix.

I have my fingers in some of our reloadable configuration code and one of the places I see it used a lot is basically a bespoke service discovery system. It's so stupid, and it adds a lot of overhead. There are at least three competing solutions (architectures, not vendors) for this problem and two of them were old hat before this code was ever written.

Stop it.

Reply on news.ycombinator.com