Deployment Infrastructure for Continuous Deployment at kaChing

Fewer and fewer. As soon as an issue gets by our testing, we get the person who shipped the bug, and a few of our testing gurus together to figure out how to add a test for it. We've built some really impressive testing infrastructure, which we're working to open source. If I was forced to put a number on it, we may see a small issue every other day, and a larger one roughly once a week.

As soon as we rolled out the Deployment Manager we saw problems staying on production for shorter and shorter times. We see the DM as more issue mitigation than testing. The self test production framework we're continuing to build out has been hugely helpful.

The blog post is mostly about backend code. At the moment we're using selenium in more of a monitoring role (think keynote but free). We need to get selenium up and running as part of frontend automated testing, but right now we're pretty reliant on Test::Unit and some basic integration tests.

Yes, where we feel it's the best solution (our standard key-value store is voldemort). SQL changes are peer reviewed and applied with a set of scripts we have to make the process easier. This isn't integrated into the deployment service yet, but it's something we've talked about.

http://news.ycombinator.com/item?id=411282 has a good description of something that's very similar to our SQL rollout policy:

"only add columns to tables and new tables, never to remove anything, so the database can be upgraded hot, and the old code can continue to run on it, only the new code will see any new columns/tables."

and

"Once you are sure you are never going to roll back to the old code you can wipe out the old columns."

This comment (and its author) http://news.ycombinator.com/item?id=411358 were formative in my views of service deployment.