Like the old axiom, the earlier you find a bug, the cheaper it is to fix. The Search Chaos Monkey tests run against the current state of the repository, are not very precise and are relatively expensive and time-consuming to run. I only know when a test has failed because we start getting notifications that the service is unavailable, or has degraded performance. I still need to go dig up logs to see how the fault actually impacted the system, then go file a bug against code that's already been committed. If I spend 15 minutes digging through logs only to find that an error was caused by a bug in Azure Search code that could have been caught at a lower level of testing before that code was checked in, I won't be very happy.
The monkey is there to test how our code works as the underlying cluster changes states and experiences failures. Before any code gets passed to the monkey, we should be fairly confident through rigorous functional tests that it performs correctly in most circumstances.
We've found bugs in production that chaos engineering hasn't caught. I can't give you a percentage, but the monkey is great at catching a majority of bugs where it has coverage (places like our provisioning and scaling workflow). Most of the bugs that have slipped through are as a result of lack of coverage. I haven't added any operations that test the service under heavy load (large amount of data or large number of requests/second), and I feel like we've seen more production issues in that area as a result.
I was wondering how many engineers are actively working on your search chaos monkey. In my experience it takes a lot of man hours to get something such as this up and running, especially taking into account stabilising the azure search product while bringing the automated functional testing up, stabilising the testing system, and then bringing the chaos monkey monkey up and stabilising all three (product, testing system and chaos system).
Again thanks for the write up.
I mentioned this in a comment on the blog post, but the main reason I was able to put this together by myself and do it relatively quickly was that I was able to leverage pre-existing Azure Search infrastructure.
We have great tools for monitoring clusters, a proven queue-based job process and a good deployment workflow. Not only does it make it easy to set up systems like this, but it also allows us to smoothly and efficiently manage an ever-increasing number of customer services. It's been a good investment for us.
Before we went into production, we were having problems with leader election that could cause multiple masters per cluster. This is an extreme chaos state not only because it could result in corrupted or missing data (as a cloud service, we take data integrity very, very seriously), but also because it didn't raise any immediate notifications, only downstream errors.
This problem has since been fixed, but even before we did that, the highest priority was to ensure our system detected this state as quickly as possible. We added a service that monitored cluster leader status and that quickly alerted us whenever anything was unexpected.
Even after fixing the problem, I've set the Search Chaos Monkey to injecting failure into leader election at various points to ensure that the cluster can always recover in an expected way that won't put customer data at risk.