Comment by mrdonbrown | Hacker News Reader

Comment by mrdonbrown | Hacker News Reader

As my startup [1] is in the domain of CI/CD, I've been doing a bunch of customer development interviews to better understand how teams deliver software. I was also surprised how few teams use full Continuous Delivery, even at cutting edge tech companies. It is indeed often used by small teams, even within large companies, where they deliver internal backend services.

The most common seems to be auto-deployed to staging or a dev environment, with some sort of daily or weekly process for promoting to production. One company built a Slack-based approval process using +1 or -1 reactions, and another has a zoom meeting where every author has to attend and is walked through a checklist before the release is approved.

My team also had a manual approval step to production, which theoretically meant the dev would check logs, dashboards, and alerts before approving, but in practice both with us and teams I talked to, that is followed about 50%-80% of the time.

What we built into our product, Sleuth [1], is a way to automatically promote staging releases when the staging release was determined to be healthy and soaked for a minimum amount of time. This allows the 80% case to simply flow through to prod without developer babysitting, whereas we can easily interrupt the process with a -1 reaction in Slack if it needs more manual testing. I think this is the ideal - the common case is the code flows but you still have an easy way to interrupt the process when the change needs it.

[1] https://sleuth.io

All those years it has been on focus, I still don't understand what problem CD is expected to solve.

Yes, there is value on CI, and there is plenty of value on very low effort deployment. But, except from a large downwards risk, what does automatic deployment bring in exactly?

To me, the main value of CD is ownership. The developer can own a change from beginning to end, and most importantly, its impact on production. That feedback loop creates better future changes and a better developer.

Automatic deployment takes it to the next level where it becomes so easy to deploy that the dev starts fixing things they would have previously ignored because it was too painful, annoying, or time consuming to deploy before. A typo here, a refactoring there, and now they can fix, push, and go back to what they were doing within minutes, but do so with the confidence they won't break things. It is kinda like the difference from when CI takes 3 minutes to 10 minutes. It isn't much but that longer time means you alt-tab over to reddit or whatever, forget what you are doing, and now that task takes orders of magnitude more of wall clock time.

Ok, I was not explicit enough on my question. The benefit of low effort deployments is very visible.

What benefit does have a computer deciding that a version of the software is ready to deploy instead of the developer pressing a button when he thinks the software is ready to deploy?

Right, got it. I think it is the lag time involved in the pipeline. Once a developer clicks merge, CI has to run, an artifact has to be created and uploaded, then usually a deployment to dev or staging is kicked off. That takes somewhere between 5 minutes and 2 hours in my research. Then, once done, the dev can click a button to push to prod, though often that is just to a canary, so another 5-30 minutes, then repeated for other prod environments. That whole process takes a lot of babysitting, and realistically, the dev has started on another task and forgot about it, making it take even longer.

By making it automated, the dev can immediately start on a new task and the deployment can go out quicker, meaning the next dev can start on their deployment. Therefore, I see it as helping preventing costly context switching and therefore, improving efficiency.

Thanks. That makes a lot of sense.

Yet, it goes completely against my experience, so you've left me wondering what creates that difference and suspect it comes from organizational differences.

You certainly considered no post-deploy activity. That should have been obvious from the beginning, but I just didn't think of it. I suspect the post-deploy activities I'm used to go to the role of a product manager, and you are assuming a more jira-oriented environment than I was.

You are probably assuming larger software blocks than I am. I should have imagined that too, because I have a tendency of breaking software in many more independent pieces than it's usual, and of imposing that on my environment everywhere I go. I am used to people having something to do on a different project than the one they started a CI, so they can start right away, and come back after the CI is complete, so there is no wait. And there is no "next dev" waiting, because he is working on a different project too.

Also, having a different project to work on during CI means a major context switch if people need to fix anything on the previous change they made.

So, with that said, I can now imagine what kind of environment CD makes sense. It's somewhere with a small number of projects and a large team of developers (or, any place with a high developers/projects ratio). It's the kind of place where would be hard to coordinate a release if it was manual too.

Reply on news.ycombinator.com