Comment by vemv | Hacker News Reader

Comment by vemv | Hacker News Reader

Parent Full thread

There's a number of startups doing some variation of this.

What many don't seem aware of is that plain pull requests, in combination with CI, entirely kill the need for a deploy app/bot.

This is how I do it at my current company:

  * use plain git flow (master/develop, hotfixes, etc)
  * use additional explicit branches per deployment target (e.g. master-spain for http://myapp.es, master-mexico for http://myapp.mx).
  * Protect these branches using github/bitbucket 'protected branches'.
  * open a PR from master to master-spain for performing a deploy of said target, detailng nicely what is being deployed and why.
  * instruct CI to deploy my app on each build of master-spain. master and develop are never deployed.

This setup has the same benefits (and then some more) than competitors:

  * Explicit deployment authors, reasons, timestamps
  * Impossible to deploy red code
  * Impossible to deploy code not in master
  * Impossible to deploy concurrently to the same target

Hope it helps someone!

The biggest issue I have with this model -- and git-flow as an opinion about how deploy works in general -- is that it doesn't take any account of rollbacks, and it doesn't scale well to arbitrary numbers of deployments.

The history of my production deploys is not monotonically forward: if something breaks, it rolls back. Nor does production roll forward as one piece: different components of the stack roll in separate motions (the db schema vs the frontend servers for example).

Git, tied to the project dev history, does not well represent these things. Reverts in the deploy branch are not semantically identical to rollbacks in production: and it's not necessarily safe or wise to merge them back into dev history.

A separate git repo, referencing release numbers, or dev repo commit hashes, would work pretty well on the other hand...

I'd say rollback is possible under a 'git driven' workflow.

That said, sincerely I find rollback one of those inherently complex ideas:

- Rolling back assumes going back to the previous commit will fix everything, an unproven (unprovable?) hypothesis in the face of database migrations, job queues, etc.

- Making database migrations reversible can be nearly impossible (particularly at scale), aside from a significant engineering effort (for something that should absolutely never happen)

So I just don't contemplate the possibility of rolling back a deploy.

Instead I try things (particularly migrations) on staging rigurously:

- staging environments always ephemeral - created from scratch for a given relase

- always load fresh production DB into staging

- check that all my model objects are still `.valid?` (http://api.rubyonrails.org/classes/ActiveRecord/Validations....) after the migration

- leave staging running a few days.

- if you really can (not easy), forward production traffic to staging as well.

If things go wrong (which under my proposed discipline would be a massive screw-up), then the fix would require analysis, a regular fix (no time travelling), and a regular deploy.

Reacting instantly (i.e. without analysis) is kind of delusional thinking. I'd rather stay broken a little longer for avoiding further complications!

I heartily agree with your sentiment "rollback one of those inherently complex ideas" :) It's true that sometimes it's not even well-defined, such as for database migrations.

Some of this also depends on context. If I'm shipping a single primary deployment of a massive fairly monolithic SaaS product, I can do this time-marches-on stuff. If I'm actually shipping shrinkwrapware -- and as a sibling comment says, doing rolling blue-green deploys also looks like this, if briefly -- switching something back to a previous code version is very worth minding.

Code rollbacks are about immediate mitigation, not about pie in the sky snapshot rollback. If you are sane about deployment and don't go to 100% of traffic instantly, then halting a broken deploy and rolling back is certainly better than shifting into analysis mode.

> I'd rather stay broken a little longer for avoiding further complications!

Unfortunately analysis is slow and an unbounded process, and high leverage businesses where every second of downtime has actual measurable loss simply cannot accept this trade-off.

Good point. Few solutions are apt for every scale and every business!

OTOH, if you really essay a given deployment again and again, you can become really confident that the operation will succeed in production.

Real example: the most important feature I've developed this year has been put 5+ times in staging across a couple months. Every time I've asserted all kind of stuff, gathered feedback from the business owner, etc.

The deployment going bad in production is just not a possibility.

At a larger scale than mine, I would probably introduce 'dark launching' as well. That would further reduce the possibility of needing rollback.

Could that not just be shuffled into the CI side of things

issues occur -> redeploy last-1 deployment

I use a very similar system to this one, but instead of building from environment-specific branches, we branch from a release branch.

We use a similar git flow system, but we build when a deployable bit of work has been merged into the release branch for this sprint of work (e.g. release-0.1). Once this merge has occurred, a build is triggered, and is then sent to our deployment tool (Octopus Deploy). Our branch naming structure partners with the build number for this specific bit of work to create a build number that matches semantic versioning.

From Octopus, we push these releases from our automated dev deployments up to UAT, staging, and eventually to live. Once we have pushed a release to live, the release branch is merged into master, making that release our new master build. When we move on to the next sprint/release cycle, we reset the build number and merge into a new release branch

We were keen to use this approach as it allows QA and non-technical people like project managers to deploy releases rather than developers. It was a bit weird to move to a workflow where you never actually deploy from master, but it's worked really well for our use case. It's also reassuring to know that the build that works on your dev box is the exact same build being deployed to your other boxes.

With that aside, I definitely agree with the idea behind your post. These are problems that can be solved with existing tools. You can do it using plain-old pull requests, or using a dedicated deployment tool. These tools already have plugins that you can throw into your favoured chat platform, or through email. It seems like there are an ever-increasing number of build/deploy solutions out there that seem to do basically the same thing, but slightly differently.

Please be aware of the gotcha of protected branch: https://help.github.com/articles/about-branch-restrictions/

Many teams - why, I don't know - dislike automated deployments, but do like being able to trigger a deployment without needing to setup local deploy environments. In this case, having a bot is useful.

In a way, my setup doesn't automate deployments - i.e. you never inadvertently deploy by committing some code or merging a PR (when your intent was merely approving some code).

That's why I made the master vs. master-xxx distinction: deployments are an explicit action, which you are free to defer as needed.

So why is a bot better than hitting the build/deploy button on your CI/CD server?

An issue I have with these git driven deployment process (not your specifically which looks well thought-out) is that your are starting from source each time.

I like my deployment across environments to use the exact same binaries that have been validated. Is it just me that is afraid of introducing issues by not doing that ?

Long shot, but you might be interested in http://repeatr.io/ -- it's a tool for our brave-new-container-world intended to make it easy (or at least easier) to suss out reproducible builds.

Or to answer your direct question: "no", it's definitely not just you who's afraid of that :)

Thanks for sharing!

Reply on news.ycombinator.com