Hacker News Reader

Comment by corty | Hacker News Reader

Maybe a stupid redundant slightly offtopic question, but I'll ask anyways:

What is the best way to couple Ansible with some git repository to automatically apply changes while still being able to trace and debug?

View on HN

Tower? Open sourced by Red Hat as AWX, it is a central Ansible hub for exactly that purpose.

Tower/AWX are bad products and they should feel bad. It’s actually easier to run Ansible via any CI system than it is to use Tower. I get what the devs are trying to do but Tower wants to own so much of the process that it’s basically not Ansible anymore. You pretty much have to write your code specifically for Tower (which means a total rewrite in mosts shops because who starts with Tower?)

That type of criticism is a bit broad. While it has all the hallmarks of a commercial product trying to fit every type of customer, on the other hand the whole point is having a centralized architecture for Ansible. This is something desirable as soon as you outgrow using it for ad-hoc jobs, and there is a payoff compared to just scripting in bash or dockerfiles. Of course the code has to be adapter for it, but it's not anything like a complete rewrite.

Ansible doesn’t have the right model for painless auto application, because it’s only idempotent in some cases and not others (e.g. removing config).

It’s a good tool to set up a server from scratch to a desired state, but not to go between two versions of desired state. So a git-based deployment workflow is doomed for failure as soon as you start actually using the history (or removing config).

We've been using the git module to pull down the desired version of the repo and then using synchronize to copy it over. That could be used to overwrite whatever was previously being copied from the git repo. So, can you explain where that doesn't work for a git based deployment? It's a pattern we use a lot. (A slight tangent though, systemd-tmpfiles can ruin your day if it partially deletes the repo, because that makes the git module crap out.)

If I understood your use case correctly, I think you’re deploying an application with it? I meant more about server configuration things with Ansible configuration which is version controlled and auto-applied on push.

In that case, say you are using Ansible to set up a daemon A on a server. Now you want to remove the setup for daemon A and move to daemon B. Just changing the configuration is not enough (since nothing stops daemon A and cleans up it’s configuration). And after you’re in the new state (let’s say daemon B was just an experiment), if you decide to revert the commit that added daemon B and push, nothing will stop daemon B.

I'm still not seeing the difficulty with doing that. Basically any daemons that you're setting up using distro packages or systemd service units (which, even though I'm not strictly in love with systemd, is what you should do for any service daemons you set up), it's just a matter of telling the service/systemd module and probably the copy or template module that you used to created the config files that they should be "state: disabled" or "state: absent" respectively. It'll disable and remove config of whatever service you've set up. What I will also do some times is create an array variable of services to enable and services to disable, and all that's required to switch is moving a daemon's name from one list to another. So, unless you're doing something outside of the modules like copying a bunch of stuff with the shell module and launching these putative daemons without some supervisor like systemd, it's pretty trivial to tell Ansible to reverse whatever it did. And if you are just launching processes using a bash script or something, why? It can get more complicated if it's e.g. a dev box and people are using ssh to go in and tinker directly, but generally Ansible will still just remove/create whatever needs to be there if it needs to be changed, or do nothing. Depending on how robust you want to make Ansible against people doing unexpected things, you can use trap doors like "meta: end_host" to bail out on errors if people are causing problems with manual edits, but that's better solved as a people/process problem than trying to make Ansible handle all possible unexpected situations.

In the end it's also usually just easier to use Ansible to bake a new VM or container configuration and deploy that instead of mutating an existing one over and over again.

If you move daemons from enabled to disabled, then you’ve taken an extra step. The difficulty is not in remembering to do it once, and daemons are a simple example. The difficulty is in always remembering, especially for trickier changes. But sure, if everyone always remembers, then it’s fine :)

I don’t think reversing what you just did is as trivial as it should be. For a git-based workflow to work well, it should be as simple as reverting a commit.

Setting up a VM from scratch is where Ansible is great. A CI/CD that recreates the VM and applies the config would work really well with automatic git deployment, but that’s generally not how people use it.

People have been chanting cattle not pets since before Ansible existed, so if people cargo cult it without using it in the way it's useful, there's not a tool out there that's going to really protect people from lack of due diligence. We don't judge a kitchen knife according to the people who try to grip it by the blade. Dependencies outside of an app's git repo will always exist for anything beyond the most trivial application.

Tooling can assist process, provide guard rails and reminders, but it's never going to replace remembering some important things. Having layers of process that mean that a single person forgetting is not going to cause disaster are necessary no matter what. Maybe when general AI exists we won't have to worry about remembering things, but then at best most of us will be out of jobs, at worst we'll be running from kill bots.

Don't get me wrong, there are plenty of design decisions in Ansible that regularly annoy me, but I've never expected it to make me less forgetful or substitute for people cooperating and communicating, and I think expecting any tool to do that is a road to disappointment.

This definitely isn't the best way because it's far from painless, but we use GitHub Actions to run some Ansible playbooks. When there's a problem that needs debugging we add a tmate[0] session with mxschmitt's action[1] so we can step through the playbook, etc.

[0]: https://tmate.io/

[1]: https://github.com/mxschmitt/action-tmate

Or Jenkins pipelined steps which might include setting up the pipenv to include the setup of Ansible in your git as requirement file. So it starts with Groovy and switches to Ansible etc

Build/Buy a real CI/CD system for this TBH. Once you hit a certain level of complexity, this sort of deployment strategy will mostly just amplify pain.

For small deploys you could set a cron that does `git pull && ansible-playbook`. You can setup your cron to email failures. This can be setup with the ansible playbook and bootstrapped by running over ssh instead of local. Cron can email failures

For more than a few servers, other answers are probably better

> set a cron that does `git pull && ansible-playbook`

AKA ansible-pull, which can greatly simplify the auth story since the machine only needs outbound access and auth to the repo (if required): https://docs.ansible.com/ansible/2.10/cli/ansible-pull.html

Didn't realize that was built in to Ansible. Depending on how ephemeral your hosts are, you could scale this to quite a bit of hosts and use host-reported health checks to determine if things are working correctly e.g. HTTP POST a "this went ok" to some web server or metrics/monitoring service

Reply on news.ycombinator.com