Why is ssh even needed on distributed clusters? Shouldn't provisioning the cluster be automated and the nodes be immutable by design? I can only imagine what a nightmare a huge fleet of special sniwflake machines woukd be to manage. Cattle not pets
Some random user with some edge case will put one of your nodes for your service into a bad state. Throwing away the machine will just lose the state and push the user somewhere else.
Also centralized log and metrics stores get real big and expensive really fast. Sometimes you simply can't afford to ship everything to it. So you'll find yourself putting detailed debug logs on the "free" ephemeral/local disks while info logs go to your central store.
If not, well congratulations, but if I got told I can't get the above to debug things I'd start question why this "pet" infrastructure lacks basic debugging ability.
What if the bug is corrupting data or sending incorrect results to the client? Even if you detect and kill the instance, you still have a problem to fix. And even if you can cleanly kill the instance and redirect the request, you can't avoid the latency hit from having to re-process it.
You can also spin up a canary machine to "live" debug.
You can reproduce the code and data, but how can you reproduce the exact state? You can't log everything that happens in a machine.
PS: How is your automated provisioning system reaching your cluster if not by SSH?
Not sure about moondev, but Terraform + Cloud-init + Container Orchestrator means that I basically never need to SSH into my nodes, except in extreme/rare circumstances.
So you do need ssh after all?
I guess that's an offensive thing to point out judging by my score...
>I said "I basically never need to". Not that I never need to. But yeah, I basically never, ever need to
It is interesting to imagine why this innocuous comment has ruffled so many feathers.
Saltstack is either using SSH to communicate or opening it's own port. I'd much rather trust an open ssh port for securely provisioning/management than allow any other piece of software to keep a port open (upto and including TLS based protocols).
This allows me to have one central server that is well secured and protected that allows ingress from the remote hosts, and then all the clients reach out to the master to get their tasks.
I build machine images with packer that get provisioned during the deployment pipeline. That single machine is then put into a cluster with x number of copies. If one dies I don't care, the cluster provisions another automatically.
A) A hardware issue (thus requiring hardware maintenance)
B) A software issue triggered by the particular workload of this machine (thus requiring a software change)
C) A network issue (thus requiring network maintenance, possibly phone calls to ISPs, etc)
D) Something else (thus requiring who knows what)
The most normal way in the real world is, have a sysadmin and a dev sit down together, SSH into the machine, and poke things until they arrive at the root of the problem (possibly with additional process involved if it's a really big system).
Then, through normal (non SSH) processes, perform the required maintenance and update your monitoring system so that it identifies problems of that class.
So not all of the nodes can be immutable and stateless.
With real, physical machines; you could have a cooling issue (which you should fix), a manufacturing defect (which you should RMA), a networking issue (which you should fix). All of these can easily be unique to an instance, and all of them need analytical attention; especially when you're paying for all of those machines "horizontally" and "at scale".
In practice, you'll of course actually be using a computer with RAM. Most likely, it will even have a Von Neumann architecture. That leaves you open to everything from race conditions to hardware failures to cosmic rays putting a machine into a bad state.
* all the state needed for serving requests is transient, not something that needs to be persisted
* any particular instance can disappear without affecting the observable behavior of the system
* the system is infinitely scalable: it doesn't matter if the cluster is comprised of 1, 100 or 1000 instancesNobody calls the computer immutable/stateless. Immutable/stateless is an architecture pattern, implying that nothing important is stored or shared between calls.
It doesn't say that you can't cache info or write log files, etc.
> Just a bunch of buzzword nonsense
Well, what's the alternative that you propose? Should my HTTP requests leave lots of state-full files laying around? Is it easier to manage deployments when every box has different files laying around depending on how old it is?
Also, remember these techniques are for people "at scale". If you only have 1 or 2 boxes, these techniques may not be useful to you, so feel free to ignore them. I've got a 1000 boxes, and have seen first hand why it's a bad idea for each box to be able to be a unique snowflake.
Even if you have physical machines, you shouldn't put your production image on it to debug it -- you should put a debug image on it that allows SSH. If it's a hardware problem, it will be easy to find. If it's a software problem -- well, you have 1000 other boxes running that software, so you'll see it again. Just add better logging.
We aren't talking about using SSH to go in and fix a machine so things stay up. It's about figuring out why something is happening.
When you have 100 machines, you can probably afford to do this. It would be silly, because when you only have 100 machines, most of your problems will be software or network issues. Still, you can do it and it probably won't cost you a kidney. When you have ten thousand, you can't.
And this is even assuming that your monitoring system notices the problem. All too frequently, you'll notice the problem in some other way. Monitoring systems are designed to detect the things you knew might happen, and problems similar to those. There will be blind spots, it's just unavoidable.
https://news.ycombinator.com/item?id=11746425 https://news.ycombinator.com/item?id=12482212