Hacker News Reader

Comment by yalooze | Hacker News Reader

I wonder what the reasoning was for having version 2 only terminate instances (vs burning up CPU, taking disks offline, etc.)? I assume it's something to do with what Chaos Monkey is NOT trying to solve (ie. eating up CPU is caught elsewhere by another system and out of scope for Chaos Monkey now). Just trying to think it through...

View on HN

See my other reply in this thread for our reasoning. https://news.ycombinator.com/item?id=12744567

Was wondering the same thing... I know in our environment, unexpected events on a box cause more problems than entire server failures. We design around servers coming in and out; specific processes failing in random ways is harder to design around.

If you can handle a server failing and you have good reporting, you can handle many of those random issues by simply rebooting the affected server. So I can see them taking out things like "load up the memory" or "load up the CPU". Logic errors (bad RAM, corrupted packets, high packet loss) are another story, but I don't know if V1 did those.

Seems to me those kind of issues would be a good way to test your monitoring. For instance verifying that the appropriate people are notified or automatic action is taken when within a reasonable amount of time after the CPU on the box starts spiking.

We have another system for outlier detection that can kill instances that start behaving badly in terms of CPU, response time, error rates, etc.

http://techblog.netflix.com/2015/07/tracking-down-villains-o...

I would assume that terminating is easy via the AWS API, whereas some of the other things need a process on the instance. You shouldn't really be connecting to boxes directly over SSH if you do DevOps correctly, so maybe they blocked port 22 to enforce this.

What do you mean by "do devops correctly" to avoid SSH on boxes? (I'm a developer, not devops.)

"Devops" has bazillions of meanings, but avoiding (human) ssh to production boxes is a generally sound principle these days because our infrastructures are becoming harder to understand by poking at boxes one or two at a time now even for forensic analysis.

So logging in to a server to check a logfile (assuming i dont or cant do centralized logging) is considered anti-devops ?

Edit: Sorry responded to wrong parent, sigh.

It's just a matter of scale. If you are at the scale of netflix, vm are probably too complex black boxes, and logs output somewhere else anyway. Plus the problem may involve the interraction of several vm, or the network, or other composants together.

At this scale, you basically have to have centralized logging. When you have thousands of parallel instances of a single application, searching logs box-by-box just isn't practical.

Consider also that if you're elastically scaling EC2 instances and you need logs off an instance that's since been terminated, too late! That disk is gone. So again, you need a central log service.

Parsing logs isn't the only way to troubleshoot.

It's not even the best way, just sometimes it's the only way.

Immutable infrastructure [1] is preferred. If SSH is detected, you must assume something on the server changed, and has deviated from the baseline.

[1] https://www.oreilly.com/ideas/an-introduction-to-immutable-i...

If deployment is automated and "clean", images get baked into machines and they just start. For instance, we use Ansible against the machine itself on boot, so we don't really need ssh access to it: everything is automated (but we keep it open to troubleshoot anything that may happen)

Everything goes through Spinnaker now, which in turn supports all clouddriver provides including aws, gcp, azure and kubernetes. Resource limits should be set by instance type. It's more of an application level thing than infrastructure which is what chaos monkey is supposed to simulate

I actually agree with you about immutable infrastructure, as I work at implementing it.

But that's dangerously close to the "One True Way". Which is certainly not the case - so much of this is evolving, and a wide variety of situations and circumstances.

To test a distributed system, there's not a lot of value to simulate all the many different conditions that can happen on a machine. From the system's perspective, you don't really care what happens on a single computer.

The conditions you need to simulate are (a) the machine being abruptly gone or (b) the machine still accepting requests, but being very slow in returning them and maybe (c) machine returning incorrect results. Seen from the outside, everything that can happen is usually (a) or (b), and with ECC memory and reasonable software (?) hopefully never (c).

I think it could be because it doesn't provide the value to the development teams. Why chase a memory leak or a CPU bug if it's just being caused by your fault testing app?

To prevent the negative effect of random machines dissapearing though.. that's a challenge that involves good ops, devs, even UI/UX I would imagine, and closer to something that users experience negatives because of in real life.

Reply on news.ycombinator.com