Hacker News Reader

Comment by ebikelaw | Hacker News Reader

This kind of thing is foreign to me because I can't imagine writing an application, to run in the cloud (as I always do) where I care if a single instance has some downtime. I always rely on something at a higher level proxying my traffic to the application, and when I need to push my application I just shoot each instance in the face and start another one elsewhere, usually on a different host:port. When I'm using the cheapest cloud capacity I can buy my application is constantly preempted by higher-paying tenants anyway, so I have to write my systems as if any instance will suddenly stop. Once you have that problem you can forget about graceful in-place restarts forever.

View on HN

I can update Erlang programs _while they're running_ without any downtime. We do this often at our company.

See http://erlang.org/doc/design_principles/release_handling.htm... for the basics. With no shared state, and no global variables, and no side effects, you can update functions so the next time they're called they're something else.

Erlang is badass in this way

> Once you have that problem you can forget about graceful in-place restarts forever.

Sure, unless you're the one running the load balancer or the service discovery system that enables other people to just kill their instances without worrying about graceful restarts.

See my other comment on this thread. Even in that case you don't have to care.

There are tons of applications where rolling upgrade doesn't necessarily work out.. anything that is even remotely latency sensitive or state-heavy, for example a game server or an order matching engine

Every service that Google operates is released with rolling updates and is fully preemptible, including the most highly latency-sensitive services and all of the stateful storage services. Google is an existence proof for the applicability of rolling updates to services of all kinds.

Not every service

For state heavy applications, you want graceful restart even if you can tolerate hard failures.

I would say this approach is for situation where the model you're using isn't always applicable.

In the case of this article, it's discussing teleport, a drop in replacement SSH server. For many users, this is their administrative connection to a fleet of servers (teleport can operate as a transparent bastion, as well as on the end host). So in our case, it's a desirable feature to not interrupt existing shell sessions, while replacing the software serving those connections. This could mean the admin's own connection to the server doesn't fail mid-upgrade, or that hundreds of internal employee's don't groan because whatever they were doing that used a shell just got dropped and they need to restart.

Disclaimer: I work with the author of the original article, and contributed to the article.

I don't take any issue with treating this as a special case scenario, although I think the modern trend is to treat the need for long-lived SSH sessions for managing services as a design smell.

What I do take issue with is the suggestion that handling such special cases is "table stakes." A little humility goes a long way in bolstering credibility.

You know that some people work on servers that are stateful? Not everyone use REST type of transport where a failed node is ok because the lb will route traffic to another place. Your solution doesn't work for things like game servers.

Presumably you just drain the servers running the old version as games end and migrate traffic to the servers running the new version?

Even stateful services like game servers could be organised to handle losing a node. It's harder to do and you need some state replication in place. (versioned state if you want to do upgrades that way) It may not be cost effective to do so. But the possibility is there.

I worked on a plan to have live migration of multiparty phone calls between data centres in realtime. The tech is available. But your planning outcome may be: it's cheaper to make single node more stable and deal with the occasional failure.

"Stateful" only means you need to care about the state being moved. Things like Java hotswap, Erlang live code reload, and your custom solutions can handle that.

Do you run game servers? They absolutely have failover capability.

I agree. The characterization of live reload capability as "table stakes" for availability is nonsense. High-availability designs never require singleton services.

Being able to replace vulnerable singletons that require process and TCP socket gymnastics with multiple instances having good fault isolation properties is the real "table stakes" for modern highly-available service architectures.

My usecase is that I have 1500+ raspberry pi class machines running my daemon process (golang agent). I want to be able to upgrade the binary, without downtime.

Next time consider using Elixir Nerves. This capability is built in to the deployment framework.

Thanks for the suggestion. It seems like Nerves needs a whole firmware to support it. These are machines where the most I can do is copy a binary onto them (crypto mining hardware). golang was the simplest solution for cross compiling I could find at the time.

Reply on news.ycombinator.com