The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change.
Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your systems. It's especially critical for disaster recovery (a DR plan that's never tested isn't worth the paper it's written on).
Consider going one step further and randomly injecting artificial failures into production shards, so handling them becomes a regular affair. When you build out monitoring, that might be a good time to think about the kind of stimulus that would be effective at exercising those monitors/metrics/alerts (think unit testing analogy). You can automate an evil adversary bot, or have humans do Red team / Blue team challenges. Yes, you're limited how much you can break without too severely impacting production, but if you've designed enough redundancy you should be able to achieve reasonable coverage. The more you can engineer simulated failures into your regular workflow, the less of a big deal it'll be when real ones occur.
Instead of watching the watchers, engage them with stimulus that keeps them sharp and prevents them from getting bored.
All those man hours cost money. Is it worth it? Depends on how much downtime costs you.
That caused quite a few problems...How much is it worth to them after the fact? I'd say a fair bit.
But hindsight...
If you are running a communications platform like Slack or Gmail, it's hella worth it.
...or just "availability testing" or simply "testing".
Yet, testing systemic failure modes at scale is way more tricky than shutting down some VMs or some network devices.
For example: saturating the uplink bandwidth on a whole datacenter.
Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question — it'd just demonstrate that without it running HA proxy's view of the world goes stale, which anyone would expect. Triggering the bug would require reducing the number of HA proxy slots below the number of webapps running for many hours. This is clearly something chaos engineering could do but IMO it's highly unlikely anyone would think to do this. If they thought of this they would also have thought about adding tests that caught such as issue long before the code went into production.
In my experience chaos engineering is often only as good as the amount of thought put into the things it does. Killing processes here and there can be useful but it often won't expose the kind of when-the-stars-align issues that take down infrastructures.
It looks like a classic lack of monitoring, as the article says. Alerting on webapps > slots, early exits, or differing views of the number of webapps up would have likely caught this.
No it won't. But it would uncover their missing alerts for a critical platform component. Their issue was exacerbated by the fact that state-management kept failing for nearly 12 hours and no one noticed.
I'm not totally against chaos testing. I just haven't seen it done well and think it's actually pretty hard to pull off (particularly the non-technical aspect of convincing people it's okay to let this thing go mad). I'd love to see how effective it was within Netflix.
It seems like it would take a superhuman level of foresight to catch this scenario for chaos engineering.
Long story. Incompetent (and dishonest) IT person. In fact, I was attacked for “harassing” the IT person.
Nothing was done about it, until the HR DB got borked, and there was no backup.
DR and backup are profoundly unpopular topics. They tend to be expensive, and difficult to test. They also presuppose a Very Bad Thing happening, which no one wants to think about.
I have scars.
I now have multiple layers of backup, and keep an eye on them. I can’t always test every aspect, and have to have faith in cloud providers, but I make sure that, even if we have a tornado, I would still be able to recover the important bits.
Backups always work, it's the restores that have problems
We don't have any personal information in our production database, but even if we did, as long as the QA is thoroughly prevented from interacting with the outside world, it can't hurt to use production data right?
Many more people that should not have access to that data, in bulk or otherwise suddenly have access which does an end run around all your - hopefully - carefully crafted security.
Keep in mind that what you think isn't privacy sensitive may very well be radioactive when combined with other data.
Using it directly is full of dangers, not only of leaking information, but also of corrupting your backups. (Otherwise, why are you testing anything?) And deriving test data from production looks like a good thing to me, but make sure to restrict the access to the test environment and mask your data.
If you really must use real data make sure that (1) you have explicit direction to do this from those higher up in the food chain so that if there ever is a breach of your test systems you don't end up holding the bag and (2) anonymize a copy of the data before it leaves the production system. That way you minimize the risk. But better: simply don't do it, generate your test data.
In that process you would go, "Ok, the service is failed and I don't know why. How does the monitoring look?", and then you'd notice it was broken. This is best performed by the person on the team with the least experience, so they ask the most questions, which reveals more.
What's really funny is when the recovery you expected to work doesn't work, and then you have bigger problems... But at least you planned for it :)
For, let's say, a b2b saas
They notify the customers first? Like, "we'll a little bit sabotage our, well, your, servers this weekend, to find out if they fail and shutdown completely and cannot start again" :-)
It's up to the business to define SLAs and SLOs that the customer will be satisfied with.
I'll do that (some day), thanks
In the past I did both. Always emit a 0 datapoint every period + treat missing datapoints as breaching to discover if an application wasnt consistently emitting metrics. In addition a lower severity Insufficient Data alert was used to discover/validate when a meteic stream literally didnt exist (normally through misconfiguration of metric & alarm dimensions).
Docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
In this case, one check could be for max age of actually deployed load balancer config, the other could be for new webapp instances getting traffic.
It's unlikely that both monitoring checks fail to trigger, and if only one of them triggers, one should investigate why the other didn't trigger.
Of course, that requires some decent amount of awareness from the on-call engineer (and/or an explicit step for that in the playbook); far from perfect.
You have to randomly simulate failure cases of all shapes and sizes to be prepared for failure cases of all shapes and sizes.
You don't know if your plan will survive contact until it's survived contact.
https://medium.com/netflix-techblog/the-netflix-simian-army-...
You would need multiple watchers who watch one another. Of course, if all watchers die at the same time, you're out of luck.
This is only part of the solution though, because in order to write and test effective alerting rules, you need to repeatedly and frequently test them with either real or simulated failures, and it can get expensive.
I would have used some Heartbeatservice but thats not feasible in our environment.
You would set up an integration test to trigger the state that results in this alert.