I've not yet linked it up to the build quality (which I want to do) and I'm also thinking of making a more Windows Focused Chaos Monkey OS project. I dislike the idea of testing only that a VM stops. I want to see what happens if I hammer say its CPU with a few higher priority threads. Or simulate insane network latency, or some other networking quirk.
I worry I am only testing against very well behaved nodes disappearance.
Ultimately I'll have a nice little dashboard, which allows me to switch instance between UAT2/PROD, that shows me the available reserve for each environment. I'm toying with making some OWIN extension that allows me to control which components are loaded. For example, we have a 'chat' component, which is obviously not essential. I'd love the idea of automatically failing that. As we add more value add features, certain models and the like which are CPU demanding, these features will not have any uptime guarantee, so I'd like to automatically throttle them, in the case of a catastrophic failure (like a DC loosing connectivity).
I am looking for something 99.999%. I've got three physical separate DC sites, each DC site has 3 or more servers. We could run the entire system on one server, easily.