Comment by throwaway9870 | Hacker News Reader

Comment by throwaway9870 | Hacker News Reader

Parent Full thread

We had a weekly deploy (plus any hotfixes) that built the images for all the new VMs (we used KVM) and then when done, stopped all the services and restarted them with their new images. Total down time was about 15 sec which for our application was acceptable. The build time was a lot longer than that (3-4 minutes), but that happened before the existing services were stopped so it didn't affect downtime. If any of the services failed to build, the deploy was aborted and the existing services were left running uninterrupted.

The deploy tool was home grown and was part of the repo. Having everything in a mono repo was really important because it dramatically reduced the number of permutations of interfaces that had to be tested down to one.

These trade-offs might not be right for everyone, but for us they were fine and allowed us to create a highly efficient development environment.

Thanks. So by deplyoing them all at once, as one unit, you avoid having to deal with the huge pain of backwards/forwards compatibility.

But what would happen if 1 of them failed to start? Or start, but immediately fail its tasks? Then you would have to revert all of them, at the same time, after some has already run for a short while, potentially writing data on the new format (for example). So I guess you can't avoid the problem with backwards compatibility completely..

Or, I guess, more realistically, you wouldl try to fix the failing service fast, without rolling back anything :)

Depends. I ran into this myself a couple times over the years and dealt with it depending on what changed. What was really nice was the weekly deploy. All commits were fresh in my head so there were not any changes from long ago that I forgot about. That was actually really important. On some holidays we would not do a deploy and even two weeks of changes was a lot more to remember than one week.

So, if I knew the changes well, I could roll back if nothing in the API changed. We didn't like that, but sometimes you just need to be pragmatic. Other times an update was made that forced you to get it working. In this case services helped because rather than the whole system being down, only part of it was. So many of our services were background type work that in most cases, customers would never even know if they were down for even 1-2hrs.

We had good staging environments with a lot of reporting, monitoring and metrics. We never deployed to production code that had not run reliably in staging for a least a couple of days.

Reply on news.ycombinator.com