Hacker News Reader

Comment by danw1979 | Hacker News Reader

danw1979·

Depends what it’s stuck doing, but you might ctrl-c it and later manually unlock the state file (by carefully coordinating with colleagues and deleting the dynamo DB lock object if you’re using the s3 backend) when the outage is over.

View on HN

Thanks, this comment made it very clear to me that I never want to touch a terraform system.

TF makes API calls to the underlying cloud. If those hang, you'll have to wait for them to time out.

Whether TF can update the state & release its locks would depend on where those were hosted. If they're in the downed AZ, then ofc. it can't do that, and manual intervention will be required afterwards. I forget if you can make those objects regional when stored in AWS or not. (You can in some other storages.)

… what would you expect to happen here?

Fun fact, for a lot of providers, it'll hang on any error, not just cloud ones. I presume it's due to the gRPC communication mechanism and the terraform binary blocking until the provider answers "yes or no" to the request

I think any system is susceptible to problems like this if the underlying hardware becomes unavailable. Using dynambodb to obtain locks on s3 is a pretty common pattern in AWS development. This has more to do with AWS than Terraform.

Rarely is Terraform mentioned in any other context.

Nothing is perfect, there’s probably good reason for this behaviour … but it is rarely something that happens anyway. and you know, deleting a key for the state lock (one that explicitly tells you when and who created it) ain’t that hard or a that big of a deal.

Control+C (once!) is usually enough to cause it to abort without any ill effects to the state file. If it really got stuck and you have to kill it, then sure, you might have to mess with it a bit.

Reply on news.ycombinator.com