Comment by orf | Hacker News Reader

Comment by orf | Hacker News Reader

I spent a bit of time trying to deploy a lambda app with Cloudformation. I wanted to use a relational database, so I needed to handle migrations.

Ok, so apparently I need to write a custom Cloudformation resource to execute a lambda function that will run the migrations prior to deploying the new version of the lambda. Kind of neat that you can do that.

Except I messed up the output of the custom resource lambda and Cloudformation completely locked my deployment up for 3 hours. 3 hours. I couldn't do _anything_ - rollback, update, whatever.

Cloudformation via a CDK is interesting, and I don't hate it, but oh boy if it gets into a weird state it can completely kill your iteration loop. And the docs say something along the lines of "if it's stuck for too long contact support". No thanks.

CF does have a lot of quirks (especially stacks locking up for various reasons, or rollbacks taking hours).

I find it easiest to run migrations when an application is first starting up (with an appropriate transaction lock so other instances won't cause the migration to run more than once), this way you don't have to do a lot of devops magic for it to work.

That’s not a fantastic model, especially for short lived lambdas. In some small environments just running a migration check on boot is passable, especially if you don’t have better tooling to handle this, but I don’t want to double my execution time working out if I need to run a migration or not in a lambda.

I guess I could do the migration check in the lambda startup rather than in the request path, but it’s still not ideal especially when handling rollbacks.

I might have preferred to use SAM, which uses CF under the hood but is much more focused on getting lambdas to work.

I'd pass in DB settings as parameters to the SAM file, rendering them as env. variables for the lambdas. with DB access managed by IAM roles (so you don't need sensitive info in the lambda env).

And depending on size/scope of project/team, might have one CF for the DB infra, and another for the app lambdas.

Using IaC services means I can have both dev and prod infra, so I can test DB migrations.

And I can do this as a solo dev on a side project where I'm also responsible for front-end and the other back-end stuff plus keeping clients happy on evenings/weekends.

Reply on news.ycombinator.com