Hacker News Reader

Comment by buddydvd | Hacker News Reader

The blog post seems to imply Stack Exchange is working with the Chaos Monkey when it really isn't. They didn't really build a system that randomly shuts down servers or services. The difference is subtle but important.

View on HN

The point here is that when you know that you're living with the Chaos Monkey, your systems become very fault tolerant. Living with a monkey does that.

But even you don't embrace the concept, the Chaos Monkey is likely going to become a uninvited house guest at some point in time. In StackOverflow's case, they bought a mainstream server with a mainstream OS, and discovered that that server came with a monkey.

Think of it another way. My brother and sister in law never worried about the failure characteristics of dinner plates, so they had lots of nice stuff. Then they had a baby. All of the sudden, falling plates and glasses became something that they had to think about.

Right, I agree that is the main point of the post. However, generalizing Netflix's naming of their fault-inducing system into unexpected failures dilutes the meaning of such name. Jeff's introduction to Netflix's system is what made this post interesting (if you disagree, take the mental exercise and rewrite this post in your mind without mentioning Netflix's system). There's a disconnect between the main point of the post and the most interesting point of the post.

It's perfectly OK for people to draw inspiration and find parallels from other sources, especially when they are both inspiring and timely.

Jeff had something happen on accident that Netflix was smart enough to engineer on purpose. Once again, Jeff learns an important lesson and shares with everyone. No need to give grief over that.

I'm not trying to give grief here. I respect Jeff highly and found this article to be insightful. If people dislike my comment for its tone, I accept it. It was probably a bit overly antagonistic.

I think the post makes the distinction very clearly. First we hear about Netflix doing this surprising thing on purpose. Then we hear about the Stack Exchange guys pulling out their hair to fix a recurring problem. But "even in our time of greatest frustration, I realized that there was a positive side to all this." Where's the ambiguity?

This is where the distinction is made: "Who in their right mind would willingly choose to work with a Chaos Monkey? Sometimes you don't get a choice; the Chaos Monkey chooses you." But, the final takeaway point of the post blurs that distinction: "the best way to avoid failure is to fail constantly." Stack Exchange didn't "choose" the "best way to avoid failure" -- the problem chose them. Netflix, on the other hand, did make that choice. The article is written as if Stack Exchange followed Netflix's path and chose to fail constantly when it didn't.

This is a good point; I didn't notice this until I read this comment.

Would have been interesting if the post had ended with "so from now on, we're going to use the Chaos Monkey."

Reply on news.ycombinator.com