Hacker News Reader

Comment by jerrya | Hacker News Reader

What I find interesting & frustrating with ansible is that it is idempotent, but not everything is reversible (or is it? I am a noob with ansible.)

So how do people test their ansible playbooks if they can't easily reverse their state?

View on HN

Ansible seems to be better at declaring a particular configuration at build time, which should be repeatable and tested, and then perhaps running limited updates in a more controlled fashion later. The destination hosts should be disposable and recreated for any significant changes. If you write playbooks to change existing state frequently you have to be very careful to write in tasks that check the previous state. If you can version the roles and playbooks carefully then you can have some idea of the systems' state. For simple idempotency, like changing an app's config template and restarting a daemon, the modules have you covered, but for more complex dependencies between services it can get hairy. Some unpredictable things can happen. Example story: systemd-tmpfiles on CentOS 7 has a poorly documented behavior of skipping files in /tmp or /var/tmp that are both root-owned and read-only. It's not something in the options or config, just a default behavior (that was removed in later versions). We had a playbook that would clone a git repo into /var/tmp and then copy the required files into their appropriate place. The git module worked fine to update the repo each run, until several weeks passed and systemd-tmpfiles blew away most, but not all, of the repo files. Some files with just read permission remained. The module could handle either all the repo files existing or having the repo disappear just fine, but not a partially deleted repo. It would error out on that task. So now there's a new dependency on a changed systemd-tmpfiles config to skip that directory, and the partially deleted mess must be deleted for a run to work on existing hosts with the old config. There's lots of stuff like that that can accrue.

I've seen molecule mentioned a lot, but I've never used it. My understanding is that it will spin up a container, vm or cloud image for you, run your playbooks, validate the result and tear down for you. Seems like a lot of trouble. Red Hat maintains it, I believe.

https://github.com/ansible-community/molecule

I use a combination of Ansible and Docker. I use it for deploying to embedded platforms where sometimes you just want to reflash the board, or you have to upload a new kernel image for some reason. Or of course when a new board turns up! The goal is so that a new user can install everything from scratch with one command (which could be wrapped into a pretty GUI). I didn't want to go with storing a flash image because it's a lot of space and not portable. This setup will work on basically any dev board with minimal changes (the docker image is platform agnostic and can also be tested on amd64 as well as arm7 and aarch64). It took a while to set up, but I like using an approach which is very ephemeral - means you're forced to document exactly what you did to get the system to be in that state. And you don't worry that if you fry a board that it'll take days to set up again.

Ansible makes it easy to do mundane things like delete the default user and set up a new account, copy over ssh keys for easy access, add services and set up the docker container. Docker has the meaty stuff that takes ages to build and install, so I make sure it works and then save the image for easy download later.

What I've currently got going is everything in one repository, it has the playbooks, docker files, tests, service files, etc. I clone that using ansible on the target machine. Ansible then pulls the repo to get new changes and rebuilds the container. The container also adds that repo as a volume so that tests and other things are accessible. I don't store containers, I just spin up a new vm on boot.

I do have a "backup" playbook which installs everything inside the container when I want to test outside docker. This has happened a few times when using custom libraries that are specific to the embedded platform that don't come with the docker images (eg hardware accelerated gstreamer plugins come to mind). On the Pi life is a bit better because there are raspbian docker images anyway.

I did most of my testing of Ansible roles with Molecule 2.22 (and Vagrant/VirtualBox) which made it soooo much easier to test roles.

Molecule 3.x was recently released but there's been some major changes which "broke" my existing setup and I haven't yet gotten around to "fixing" it.

It takes a bit to learn how to use Molecule but, if you do much role development at all, it's definitely worth it!

Most of the time I'm using Ansible to build containers or AMIs (or other images), which are then deployed, and overwritten for updates.

Other times, I write a secondary playbook to roll back some deployment if needed. (Though sometimes it's as simple as setting a `state=present` to `state=absent` for a few tasks and that's all that's required.

Virtual machines.

If linux only, docker is great for that because it's so light, and it easier to compare layers of an image.

Reply on news.ycombinator.com