Ask HN: Write tests for your infrastructure automation
Ansible created the instance, installed the necessary packages, templated the configuration files, created appropriate directories, added Nginx (access to the UI), configured supervisord to manage the django web app and carboon daemon, and a bunch of other things.
Each task/step in Ansible should tell me the state of the deployment (has config file A been changed? Have we reloaded carbon? Is carbon running?). I can also use Ansible to tell me whether or not http://graphite.internal.company is accessible and whether or not I can see the data coming through. I can also write a test in Ansible by sending dummy metric to Graphite over socket (to verify that I can actually send things to carbon and the data is then saved into whisper).
I also want to know if we rotoate our logs or not (yes sure we deployed logrotate config file yesterday and my policy is rotate daily, so the next day I want to check if rotation happened).
See, Ansible can do that. I can write plugins/modules to handle all the above.
But the question is, to what extend do people write tests for their infrastructure? There are probably a lot of things I question myself after deployment. Another thing is, sure I can write everything in Ansible, but would my tests be better off writing in my favorite language, and yes perhaps run using Ansible (especially if I need SSH access).
Because my end goal is to reassure myself, and eventually produce my own Chaos Monkey. How can I run Chaos Monkey if I only have the tests for applications, but no tests for infrastructure?
Thoughts?