Circuit Breakers in Elixir | Hacker News Reader

Circuit Breakers in Elixir

blog.rokkincat.com

82 pointsuntothebreach9 comments

Open articleView on HN

So what is the benefit of the iTunes fuse example over a very simple

Try iTunes

Catch exception

Try local search

I guess in a bad state it makes your results faster. How often does that need to happen to justify the complexity?

My thought was the complexity would be worth it if you have a non-trivial number of users.

Additionally some third party API's have rules where they expect you to back off exponentially if there are errors. You'll get your account suspended if you don't play by the rules.

Another good trick is to melt the fuse if the request is merely slow. This can often back off before the timeout scenario occurs. I used this to great advantage when I crawled all duels of Quake Live years ago. If the site became slow, I would cease crawling for 15 minutes, and fuse handled that nicely.

I think the key is you need a backup for the system you are gating off. If your db breaks you blow a fuse and show errors... Not sure you gained much.

There is a subtle but important difference:

In a timeout situation, you know relatively little.

If the circuit tells you it is broken, you know that the backend system is deemed down and trying again right now is probably futile.

Each client for a server (on which the fuse runs) will be rather blind to the experiences of other clients. But the server, and thus the fuse, will know what happens to all the clients. This allows you to act globally on the server side. But this in turn empowers each client locally since it now also has global information.

So making a fuse global, say throwing it in your db, now means you can't have a fuse about the DB. Which may be okay, but seems in some ways less good than a local fuse.

Netflix has a good explanation here: http://techblog.netflix.com/2012/02/fault-tolerance-in-high-...

Disclaimer: I'm the author of fuse.

Circuit Breakers provide three things not provided by your above scheme:

First, there is a configurable policy on how many errors to tolerate before breaking. This policy is not baked into your code, but lives outside, often in configuration files. Some of the work that is currently going on is the support for more advanced policies and more advanced ways of ramping up connectivity again on the flip side.

Second, there is no resource buildup. In your example, every request to iTunes will wait and use resources while it is waiting. Once the circuit breaks, you immediately respond with an error. In a system with 10k req/s the buildup is pretty serious if you have a timeout of 5 seconds, say, since it will effectively be 50k reqs waiting. Which could be 50k network sockets.

Third, fuse has a monitoring system built in. Any fuse you create will post its current state to an event manager which can be used to build monitoring applications (essentially this is a low-volume pub/sub pattern). Rolling your own, you have to provide this monitoring yourself, but using fuse, you get a nice way to plug into the fabric. This is used by Riak's Search system yokuzuna for instance.

Finally, what makes fuse special from the other circuit breakers is that it has a full QuickCheck specification. I.e., we have a pseudo-formal account of fuse working as intended according to the specification. In particular, I tend to generate random fuse scenarios for a couple of hours before releasing new versions. This amounts to a couple million random test cases, and we approach a full model-check of the code as we spend more time generating test cases. There are some novel work in there with respect to handling randomness and time in test cases via Erlang's QuickCheck's excellent mocking system. As a result, there have been few bug reports and likewise few fatal errors reported.

(Edit: for completeness, all the code of fuse is online here: https://github.com/jlouis/fuse including documentation)

I like it thanks for the details.

Now I need to find one for python to tinker with.

Reply on news.ycombinator.com