Comment by simonw | Hacker News Reader

Comment by simonw | Hacker News Reader

Parent Full thread

Not exactly no downtime... but all site content stayed available throughout the move thanks to our handy read-only mode. We added this early in the life of the site and it's proved extremely useful for a couple of major moves. It's a pretty simple implementation (based on our site-wide feature flags) - once in read-only mode all login cookies are ignored (so everyone gets the signed-out experience) and the sign-in button is disabled. We also return an error for any POST requests just in case someone has already loaded a page with a form before we turned read-only mode on.

The moment we've gone in to read-only mode we can create a brand new instance of the site from a copy of the (now frozen) database, make any necessary changes to that, then switch traffic over to the new instance once we've tested that everything is working properly. If the new version of the site has a problem we can turn read-only mode off and continue to run on our original database.

If you're running a more content-oriented site it's well worth taking the time to set this kind of thing up - it's not too hard to do, and it gives you an enormous amount of flexibility for maintenance further down the line.

This is excellent advice! I can't tell you the number of times I've worked on projects where I wished just such a "read-only" mode existed. As an added benefit if you have a read-only version of your site, I know at least Akamai (and possibly their competitors) has a service where even if your origin server disappears, they will continue to serve the latest version of your read-only site until the origin servers reappear.

We have varnish at the front of our stack with a one-minute cache timeout for users without cookies - but thinking about it, there's no reason we couldn't bump that timeout up to something much higher for the duration of read-only mode (though since the servers aren't having to deal with writes they don't really need an extra performance boost).

Check out Varnish's configurable grace period.

It will basically instruct it to continue to serve content that it considers "stale" (up to a point) until it is able to update it.

Going one further, if you really do have to maintain doing writes, what you do is you modify application code to do simultaneous writes to both databases instead of one (i.e. "fork" your writes).

This is where you are super glad you did the right thing (did you?) and your DB layer is abstracted and your SQL is standards compliant, and this saved you hours of headaches. Anyway, you maintain reads from the old db.

Then, as the site is running, you migrate all data pre-fork, to the new database. Finally, after validating that you got it right, you flip the switch again to have reads come from the new DB.

But you're not done yet. Keep the reads forking for a while, till you're sure everything went okay. If not, you can flip back to pre-migration instantly with zero data loss.

Presumably, all this flipping between dbs is done through some kind of flag that can be modified at runtime, so you don't have to do more code deployments, because you're dancing with the devil here already.

In any case, with proper capacity planning and good code, this is also doable, but not super required unless writes are mission critical for your customers 24/7.

But the last thing you want to do is to flip the switch, pray, lose customer data.

Doing simultaneous writes to two databases at the application level is very fragile and have many points of failure, all the places involving write in the application code. Also distributed transaction is probably not in place since it's an one-off thing, and there are risks of inconsistent updates to both database where one write goes through and the other one fails. Recovery from that would be nasty since failures can be on either way.

A better approach would architect to write all the updates to a persistent message query. Then a updater can read the update messages from the query and apply them to one or both database. The updates are welled ordered with respect to time and to both databases. The potential failure scope is limited to one place and it's easier to go through the recovery cases.

Persistent queue has other benefits in scalability and fast apparent saving to the users.

Sort of. It's a tradeoff. First off, I was suggesting doing the dual writes in the code that does the write in your DB layer, not by doing fixes all over the application code. That reduces the code changes by a ton.

Second, I guess it's true that there are some consistency issues since you have to commit one transaction only if the other succeeds. But this is way smaller a risk, I think, than yours.

With your method you're writing way more code just for the update and adding a whole new layer (both failure modes in their own right), and then just moving the transaction inconsistency problem to the updater instead (when you choose to write to both databases). The benefit is that you can "replay" the updates from the updater, but the truth is not all changes are idempotent so replaying some queries may fuck things up without storing DB state (e.g. inserts, increments) which is a mess in itself.

So, I guess, choose your poison :)

Great tip about read-only mode, Simon! This should help us plan our migrations with zero downtime.

I was wondering about disabling writes to databases that are critical to application to support read-only mode with application logic to gracefully handle write failures.

I thought about disabling UPDATE/INSERT statements at the database cursor level (we have our own custom Django cursor wrapper class) but we're confident that it's not necessary with our application - we don't have any writes that we can't afford to lose which aren't triggered by signed in users going to our main application database.

Thanks for the write-up - interesting read.

Out of interest, how do you catch and block all POST requests when the site's in read-only mode without duplicating code? Not sure if you use CBVs at Lanyrd - if so, do you use a common mixin? If not, how?

We have some common Django middleware used for every dynamic page on the site which deals with that (among other things). We also strip cookies at the Varnish layer.

I believe you can do this quite easily with Django request processors, though I've never tried it.

This could work: https://gist.github.com/4066816

https://bitbucket.org/zalew/django-readonly

https://github.com/streeter/django-db-readonly

Great tip. And kudos on architecture done right.

Reply on news.ycombinator.com