Hacker News Reader

Comment by easton | Hacker News Reader

Something funny (well, kind of sad) about CloudFormation I noticed this summer was that if you deploy a CloudFormation stack which updates a ECS service and deploys tasks which then fail health checks, CloudFormation will do nothing about this and just let ECS keep killing and restarting tasks for.. well, at least several hours. You have to know to go into ECS and drain the tasks manually and then initiate a rollback from CF to get your service back into a good state. The bug reports about this I found were going back years.

The upside is that I got really well acquainted with how ECS worked.

View on HN

This support for CloudFormation across AWS services is crazy bad. For the longest time you couldn't have an ALB rules with it. To fill this gap you have to create dreaded custom resources... which can get suck for 4 hours.

CFn supports stack rollback alarms. All you need to do is create an alarm for failing ECS tasks.

IIRC, the problem was that rollbacks couldn’t start until it agreed that the service deployment had failed, which wouldn’t happen until I manually drained the task (or waited an unknown period for CF to timeout). You could click the rollback button but nothing would happen. I don’t think the alarm would’ve helped, but maybe I’m misunderstanding.

I'm referring specifically to this feature: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...

CFn will allow you to specify up to 5 alarms to monitor during stack deployment as well as a duration to monitor them for up to 3 hours.

During deployment, the stack does not transition into the COMPLETE state until the monitor duration has completed without the alarms being triggered. If one of the alarms is triggered, the stack immediately rolls back.

Since failed ECS tasks create CW Events, you can set up alarms on them and use them as the rollback monitor trigger.

100x this. Prior company committed to doing Infrastructure as Code and CloudFormation worked well except for this hiccup. We didn't even have that many services on ECS but we probably had 1 ticket a week asking support to help us with a 'stuck' stack.

Our commitment to CloudFormation was doubled down on that we could do containers, Lambda, and 95% of any other AWS Services....

However, in hidsight using SAM and the ECS CLI probably would have resulted in a more predictable CI/CD process as we weren't fighting deploy semantics through CloudFormation abstraction.

What we ended up doing is mostly decoupling CF and tasks. CF manages the infrastructure. Software deployments / tasks are done by another system which understands issues like this one.

Yes! It drove me nuts! I had a task that I forgot to give a / 200 ok and it just thrashed like crazy. I think once I have the cluster auto rollback it solved the issue though (other than ensuring a reliable health check). They should fix this!

Agreed - very painful. I think there was a recent improvement to how CloudFormation handles failed ECS deployments but it's not on by default.

I had this same bug!! Cost us like $1000 before we fixed it;

Why did it cost so much money ?

Your updating the service to a broken image? I would say ensuring it's healthy falls outside of the scope of CF responsibilities. `aws ecs update-service` won't even do that. ECS should wait till your new image is healthy before draining previous image versions. If you deployed a bad image, probably needs someone to look at it anyway.

Reply on news.ycombinator.com