Hacker News Reader

Comment by cube2222 | Hacker News Reader

You can use more than one tool.

CloudFormation is great because of its transactionality, so it lends itself nicely to deploying multiple services which are versioned together. You either succeed fully, or all services will be rolled back.

This way you can deploy your whole infra with Terraform, and then deploy to your i.e. ECS cluster using CloudFormation. Works great in practice.

View on HN

The rollback functionality of CF is a blessing. We use both CF and Terraform at my company and i vividly recall multiple times where my connection had cut out during "terraform apply" and left the Terraform infrastructure in a half-finished state.

Rollback doesn't always work with CF. I've noticed so many times that it would mostly delete everything but not certain things once in a while. Then you're left having to play detective to manually figure out what you need to delete while having to delete dependencies by hand in a specific order.

I've spent hours just waiting for CF to fail deleting EKS or RDS related resources then I end up getting billed for $30+ a month sometimes because I forgot to manually delete a NAT gateway.

> The rollback functionality of CF is a blessing

When it works, which is a big caveat: we had far more cases where it failed in a way which required manual remediation and the gaps in validation meant that you'd be in a “apply / error / rollback” loop requiring 20+ minutes before you could try again. Terraform was always considerably faster but it was especially the orders of magnitude improvement in retry time which convinced most of us to switch.

The CloudFormation team has been working on this so it's possible that experience has improved but the scar tissue will take time to fade.

This!! when you stack get stuck in error state is the worst!! Hello kinesis stream rollbacks.

It's especially interesting from a product design standpoint: everyone knows things break sometimes but when it forces you out of an easy path into something much harder or simply locks you into a lengthy penalty time delay, people will remember that one time FAR more vividly than all of the times it worked normally.

This is especially true for things like CloudFormation where the user is likely on a deadline or in a stressful situation trying to fix a problem or hit a deployment window. Forcing someone to wait an unpredictable amount of time for no reason ramps up the stress level massively.

> i vividly recall multiple times where my connection had cut out during "terraform apply"

The issue could be at least partially resolved by using automation (like atlantis for example) to apply your plans.

Reply on news.ycombinator.com