Managing server access in a multi-account organization is a real issue, though. I currently manage 11 AWS accounts and the best solution I've implemented so far is extending NSS and configuring sshd to query our identity provider (Okta) for user/group information and SSH public keys. Each type of server is configured to permit access to a subset of Okta groups. For example, members of the DevOps group get full privileges, anyone else that has a use-case for using SSH (like in a QA environment) gets some form of limited access. With this in place, I can grant/revoke privileges and manage developer keys all from one central location.
Another way to think about it is, if you're an SRE, you want to eliminate toil, and an interactive SSH terminal is toil.
- Encourages you to go to instances and check stuff rather than improve monitoring / health checks.
- Can do quick fixes on a few boxes rather than re-running the deploy. Great! But terrible when the person who knows how to do that is away.
- Tailing log files rather than centralised log management for all the things.
- Trying things out / quickly checking something in production rather than being rigorous about keeping test / staging in sync with prod.
The “problem” is ssh is such a great affordance (until you have tons and tons of instances and you can’t do anything by hand anymore) that it means you don’t need to fix internal processes and tools around deployment, configuration and monitoring.
If there’s no workaround you feel the pain and will be forced to set things up right, usually with benefits to security and repeatability.
As is often the case, the best thing about ssh (in terms of managing infra instances) is also the worst thing.
With that said at very small scales it might be overkill to automate all the things so sure, fill in the gaps with ssh and a wiki page.
I don't think it does, well it doesn't when you have > 40 machines anyway. Plus it doesn't give the ability to compare and contrast simply. (graphs are _awesome_)
> Tailing log files rather than centralised log management for all the things.
Yes, I tend to agree. But proper centralised logging is either exceptionally hard, or a hefty splunk tax. That also encourages people to derive graphs from logs, which is arse about face. Graphs first, logs when you are desperate.
> Trying things out / quickly checking something in production
I can see this, but normally one would expect people to not have general access to prod, if they are going to do that...
For example, managing ssh keys for an individual is gloriously simple, but managing them for a large organization is a huge headache. You want to use ssh certificates, but even those are implemented in a weird way, and really you should use an SSO system for auth. (This makes that easier/better, so, yay?)
When people start sshing into production servers, they end up making local changes. They focus more on the "pets" aspect of managing systems rather than as "cattle". They have to install a litany of extra software to diagnose and troubleshoot bugs, rather than expose system metrics and tightly control the app environment and its operation.
Remote access to production app servers is basically a backdoor waiting to happen, and may violate corporate security policies. When you have local user access to a Linux host, it's almost guaranteed you can privesc to root.
Finally, almost everyone I have ever seen will either force-ignore/auto-accept host key changes, or just accept them blindly, because IPs and hostnames may change, and there may be multiple environments you haven't logged in to, etc). This completely defeats the purpose behind mitm protection, which is the main intent of using SSH, though these days its other features may be arguably more of a reason to use it.
And for the tech hipsters out there: "it isn't serverless!!"
There's a lot of things that require more than easily exported system metrics and logs to troubleshoot.
While I've played around with using PCP's perf plugin to try and remotely do things with perf, generate flamegraphs, etc., it doesn't work nearly as well as just SSH'ing into the thing and running perf directly, especially if the perf data file is going to be large. I don't see how you could do serious performance engineering work without SSH access.
But, I think I'm nitpicking here, because I generally agree that there should be very little to no reason to login to servers via SSH day to day.
On the other hand, I tend to lean more towards a ‘wild animals’ model, where, sure, you can tranquilize one and bring it in to look over, but once it’s got the smell of humans on it, it’s doomed if you let it back out in the wild again.
Once you ssh into a production box, it is forever tainted. Sure, poke around in it, install some perf tools to run some diagnostics, learn what you can about its behavior. But then, rather than putting it back into the wild to serve traffic, out of mercy, you should destroy it and replace it with a clean instance.
Thanks for the super informative answer. About the quoted portion... yeah! I assume it's my responsibility to do something... like manually check the host IP or something? What is the recommended practice to deal with this situation?