Comment by perlgeek | Hacker News Reader

Comment by perlgeek | Hacker News Reader

Parent Full thread

> What are you hoping for I guess? Im genuinely curious. Maybe you want puppet to complain before running that system X doesn't have version C of package K ?

In the past months we had several failures where new features weren't working in production, and it was usually due to neglecting to upgrade some components.

So for example one developer creates a new feature in the front end, a second developer creates the backend service, notices a missing feature in a second backend service, and delegates that to a third developer.

When the time comes to roll out the frontend service, the first two developers probably remember that a new version of the first backend service is required, but sometimes forget that there's also a dependency on the new version of the second backend service.

So the feature goes into production, but fails because an indirectly dependent service wasn't upgraded as well.

We only have one testing environment, not one per service, so we don't easily notice the problem in testing.

So, if all the services where installed on machine machine, the developers would just need to declare the proper dependencies (which they can do at the time they discover them), and an "apt-get install <frontend-service>" would resolve the whole dependency chain, and either abort the upgrade if the new version of the second backend service isn't available, or upgrade it too.

What I really want is the same easy of dependency management, even though the packages aren't all installed on the same machine. I simply want to declare all of a packages versioned dependencies, and then say "upgrade service X" and it'll automatically tell me what I need to deploy (or even do it for me).

Currently our workaround is a wiki page in which we document the depenencies, and manually resolve them when the time comes for a deployment. But this is very frustrating, because the problem behind it is solved in any decent package manager -- but only in the single-machine case. So just because we run more than one machine, we have to reinvent the whole dependency management wheel.

And it also frustrates me that I can't seem to find much discussion of this topic, even though I'm quite sure we're not the only ones who have these problems. Am I just looking in the wrong direction?

Thanks for explaining. This is a pain point I think everyone that does service oriented architecture has to some level.

Most shops I have worked in seem to mostly attack this via procedure. Ie - release is OK'd by QA, then deploy ticket is made that specifies what versions go where, and then operations is responsible for executing this procedure and ensuring its sanity.

If you have a lot of hands in the pot, (too many cooks), then this is bound to have issues when everyone is busy and things get overlooked.

There is a lot of automation that can help however. Dependency management in this sense I think has to be more elaborate than just "is X installed or deployed", but you might also want to ask "is X healthy?" Just because a package has been installed doesn't mean its working.

So that is in my opinion why there is no one-size-fits-all solution for this. Many releases are fully compatible with each other, and in simple applications probably even more so, but once you have many teams working on many services, it can be a nightmare.

A lot of places have granular feature management. So, frontend can fail gracefully if feature is not supported on backend. Often backend supports more than 1 fronted version set. You can use versioned api's (like putting a version in an http header, or namespacing features like /api/v1/x, /api/v2/x). (it's been said a huge distributed app like facebook has something like 500 different versions of components all running concurrently)

You can have your deployment automation do things like check to see if backend has required version (which would be something that you would have to put in hiera/equiv of what versions are required), and even more importantly if they are responding to health checks.

One project I currently work on has a very distributed architecture, and every application has to conform to a health check endpoint that returns its state and its version. So we have scripts that do either rolling deploys or green/blue deploys depending on what we are deploying, and we can define the order in which services are deployed. We can deploy them concurrently/rolling, or independent/rolling, or either of those flavors non-rolling, or green/blue.

So if we say we are deploying fronted3.1, backend4.2, and we are replacing frontend3.0 and backend 4.1, this would be seen by the deploy scripts (and operations team) as a single changeset for the environment. So if frontend3.1 needed backend4.1 to function, we would specify a independent rolling deployment, so for each service thats being deployed (starting with the backend), we would roll out server by server at a specified interval, and we would wait for that server to return a healthy health check with the specified version. If this fails the deploy is rolled back and notifications are sent.

^ with this working correctly, we have to have the understanding with the application developers that we will never release a backend that is not backwards compatible with any components that rely on this. (beyond a cutoff version, which those do exist too)

In practice this works quite well, but is not magic, and definitely requires some up front development of automation.

So, sorry for the long post, I think this is a long winded way of saying, my opinion is that this belongs in a configuration management, as the logic encapsulating "what version goes where" gets more complex over time, and so some code, logic, decision making, and failsafes all have to be carefully thought out.

Thanks again for your detailed answer.

Currently I think the simplest way forward for us is to check versions of dependent services in the monitoring health check, so that at least the operators have a simple way to figure out missing dependencies. All other options (blue/green, rolling, ...) require too much infrastructure automation that we don't yet have.

Though it still itches me to write some generic, remote dependency management that's flexible enough to be used in different deployment scenarios and with diverse tools. Maybe later :-)

Reply on news.ycombinator.com