How to properly manage SSH keys for server access
paepper.com
paepper.com
I fundamentally disagree with this. I’ve been writing software a long time and I used to demand I have server access so I can tail logs, creating the problem this article talks about resolving (good read btw). But I can’t help but wonder why we keep teaching this mindset. We have log analysis tools available on pretty much every cloud now. Docker has an aws log driver, a gcp log driver... etc. Backend developers should make a conscious decision on how to ship log events out of the box and into something searchable, indexable, and can derive metrics from. I ran a cloud infrastructure group that vehemently said “No” to any dev requesting ssh access. Not because they couldn’t, but because we don’t have access either. We created our platform without the ability to modify it. Infrastructure as code. Only way is to redeploy the stack. You’d be surprised at how much less stress there is when you can have alerts on log events from your application when things break instead of a support call or a support ticket. Proactive > Reactive debugging. I also understand not all shops are at that level of maturity. I’d love it if the community as a whole stopped teaching people to treat their app and server as a second home, as a pet, that must be nurtured. Obviously these are my opinions and the article itself addresses how to handle ssh keys for server access in a logical way, my only issue is why create that mess in the first place?
Actually, in my company, we are pushing the logs to an external service, so I guess that line I wrote there was not totally thought out. ;)
However, it still might be useful for certain people to check servers for health or other metrics and you might not always have everything as perfect as you wish. I guess you already stated that yourself.
So in general I'd agree. Good comment!
Also, if you aren’t on “cloud”, odds are you are still using something like Kubernetes or DC/OS or Swarm or the like. If you aren’t then well, wordpress sites aren’t really in the same ballpark technically. (Joke, Wordpress sites get traffic, some lots of traffic, I don’t discriminate against the PHP tribe).
However, for anything shy of full serverless, just s/developer/ops and you still have to solve for the same problem. At the end of the day you're always going to want someone to be able to get a shell on the box for troubleshooting some critical issue. If there is a box.
> ship log events out of the box and into something searchable, indexable, and can derive metrics
That's just a way to spend a ton of money. There's really not a reason to ship or index logs. Just leave them where they are produced and instead of shipping them to a central thing that inevitably becomes an expensive bottleneck, instead develop ways to access them in place (other than SSH) and frameworks for large-scale distributed searching where you push the predicate out to the machines where the logs exist.
On the point, what do you do if the machine is not there anymore? Your model fails immediately. Also, some processes operate on an assumption that the "machine" will stand-up -> do-something()- > die(), which your model also doesn't support. Not to be messing with you, but I don't think there is the "right" model, but whatever works for us. I do like your model for immediate accessibility of the logs.
Your storage/indexing money spend scales with your company. If you have < 10 hosts you can probably handle logs with a cost that's a rounding error with something like loki+grafana.
If you have a bigger environment and more people, the cost will be more significant, but you'll also likely have more revenue to justify it.
This is the age old problem of simple problems being harder to solve at scale. Running a blog on Digital Ocean is very different to running a million dollar application on AWS.
[1] https://medium.com/@Joachim8675309/devops-concepts-pets-vs-c...
I'd rather not engage in pointless unprovable arguments, if you don't mind.
> The problem is that you are treating your logs as pets when the truth is the logs are also cattle. Virtually all debug logs will pass through their life cycle without being read, so indexing them is just a flagrant waste of energy.
You shouldn't have debugging enabled on production systems unless you have an interim process that filters out debug messages before they get indexed (and thus you can toggle which logs get indexed there rather than reconfiguring / redeploying all of your application nodes).
Also nobody is suggesting logs should be treated as "pets". You still want to purge out older logs however the problem is you cannot always replicate reported errors so if you don't have those log messages captured then you're sod out of luck.
Don't get me wrong, there is a certain allure to the traditional method of systems administration - I've been on both sides of the fence - but central logging services have so many other benefits such as security (tamper proof logs, users don't require SSH), ease of use, persistent logging, etc. The only real downside is cost but that quickly becomes absorbed in your pricing plan when customers start asking for SLAs.
By the time you want centralized logs, you are probably already spending much more than the logging will cost on infrastructure.
What you're doing here is constructing a straw man argument while agreeing with the same point I was making.
> > ship log events out of the box and into something searchable, indexable, and can derive metrics
> That's just a way to spend a ton of money. There's really not a reason to ship or index logs...
That's what I said. ;)
> One of the ways to make it seem a reasonable suggestion is the broader topic of not logging into machines at all, ever, and why that may be a good thing.
Except none of that was being discussed. Only log access.
> They're explicitly talking about not logging into machines for log access.
Why can't you just admit that you didn't read the thread properly rather than insulting all of our intelligence with these piss poor mental gymnastics where you redefine the context of what people had very clearly written.
The conversation you're replying to, however, did. I know this for a fact because I was involved in that conversation and I made that distinction myself ;)
Not only is that vanishingly unlikely to be true with this crowd it’s also not at all the kind of rhetoric that they’ll listen to either.
I guarantee that you haven't, as the number of people that can make that claim in relation to my background is vanishingly small and I know almost all of them by name.
I centralize logs. Not only does it make more sense for administration at scale, it's invaluable for security reasons and assists in compliance by providing a controllable guaranteed audit trail.
You may have a valid argument about cost here for some applications, but it's unwise to make arrogant claims you cannot back up.
Proper Information Security policy is one very good reason. If a box is compromised, you don't want the logs being tampered with.
You ship the logs to a central, dedicated server so that a clear audit trail can be established.
Anyone who needs access could remotely run a command that would return the results of whatever permitted query they submitted.
You can't hire twenty developers that all have the same skill/inclinations, the same interests, the same experience.
That's not to say that a DevOps Engineer is some super 10x rockstar developer - no, they're going to have the same variations on skill, interests, experience, etc.
It depends on your environment, but there's so much different tech once you count the entire stack, that I don't think it's reasonable to expect any one person to be an expert on all of it, or even a lot of it.
> > Every developer needs access to some servers for example to check the application logs.
> I fundamentally disagree with this.
So,
Developers shouldn't be reaching for SSH access to check logs.
If you're encountering problems that you can't diagnose through the existing logs, then you should probably be involving at least one other person - someone who has that production experience, who might have some additional knowledge about the problem.
If, and only if you've exhausted other avenues - then reach out for SSH access. But it should be a last resort, not the first resort. Plus, anyone SSHing into production boxes should really be very familiar with how production is configured. You can do more harm than good by poking around on a production box being completely unaware that you're causing alarms and outages elsewhere because you taking a memory dump of nginx caused in-flight requests to get timeouts and so-forth. The people with that experience are generally the DevOps/Infrastructrue folks since they're the ones who deal with production all day, and are going to get the pages if something goes wrong with that.
Yeah, perhaps. It'll depend on the circumstances, right.
I've got a reasonable amount of GSuite and Exchange experience, and same for Active Directory. I'm reasonably confident that I can work my way around those and do most of what I need to do without breaking it.
I needed some GSuite groups set up, some folks added to them, and a GSuite OAuth application set up and some values passed back and forth to do some integration. The only way to do all of that is with full GSuite Administrator rights.
Now, I could ask why they don't just give all Devops folks GSuite admin rights, it'd be much easier (for me) and I could do my job more efficiently.
The response is going to be something along the lines of:
> You don't need that access most of the time. > The times you do need that access it's often for a limited time or scope, and to resolve a specific problem. > For now, It's better that you work with someone who is responsible for that stuff on a day to day basis to do those things.
This is, in my opinion, pretty reasonable. Sure, it meant more delay until someone was available to do the GSuite configuration, and we needed to jump on a call to pass IDs back and forth and test it out. But it got done, and it wasn't overly burdonsome.
They didn't hire me as a GSuite, Exchange or AD Admin - they already have folks to handle that. That I have that knowledge and experience is still useful for the company - I know exactly what to request is done, and we can talk on the same level about it. Heck, if it turns out that I need this every day and I'm constantly going back and forth with them on setting things up - then I might get that access, but it'll probably come with an explicit requirement that I use it in specific ways, that I follow their processes/procedures, and keep them informed on when I'm using it and why
There is also the case of ratios. An organization probably needs more developers in specific areas than DevOps, so with dedicated DevOps you could concentrate similar work from across several teams to a dedicated DevOps team that knows that work very well.
Giving everyone production SSH experience is, in my experience, a way to run into all sorts of weirdness, not to mention endless frustration.
In a modern automated infrastructure, that box is likely a container running on a virtual machine that's ephemeral and can (and probably will) go away at any moment based on any number reasons - maybe CD kicked off a new deployment, or maybe the load changed and the instance was selected for scale-down, or maybe our spot bid for that AZ isn't sufficient for keeping the instance around, maybe you being SSHed in and poking around impacted the health-check, and so it's being killed for not performing right.
Theres many other problems, too - lots of applications are built in some way that there's simply no other way than secrets (passwords, api tokens, keys) to reach other systems, particularly third party systems. So production boxes have production secrets, which you probably don't want to share with everyone.
Giving everyone SSH access so they can, in theory, take nginx/kernel dumps as needed tends to imply giving superuser rights, which means they can do whatever they like.
So, yes, pull in someone else - find some way to try and reproduce the problem NOT on production, if that fails, perhaps there's a way to grab enough detail or pull additional logs or network captures to identify the issue. If that fails, well okay, lets SSH in - but we need to coordinate that to ensure that instance does't go away, and doesn't impact production while you do it.
But honestly, how often is a web applications bug due to a kernel bug?
I swear, sysadmins were annoying as an application developer, but devops is something else.
People with one year of actual work experience get hired as devops, and have all the privileges I would need to fix their mistakes, but I can’t, because I’m an ‘application developer’. So instead you end up teaching them how to do their job.
I’m not salty at all.
You can write the same screed full of generalizations from the perspective of any job title: a devops person would lament the fresh-out-of-bootcamp “application developers” who have no idea how systems work together so write SQL queries that retrieve a million rows, one at a time. “Works on my local!”
Saying the emperor has no clothes is not white tower thinking,
If your programmers aren't delighted with you, I'd say you the Ops person is not practicing DevOps or you have a buy-in problem to DevOps practices at an organization level.
My previous role had my title as "DevOps Engineer" but it always rubbed me the wrong way. I was just an Operations Engineer with a focus on making my developers' jobs easier, in any way I could. Having that as my North Star kept me honest about the work I was doing versus considering the role more like Operations Engineer v2.0.
In the Silicon Valley, at least, DevOps seemed to be (seems to be?) sort of in vogue; I think it's important to keep its core qualities of bridging Development and Operations in mind as opposed to just shifting an existing position's title in an attempt to attract talent.
And this should extend throughout the organization. If Architecture or Security or any other group is making your life miserable, they too should be DevOps'ing, working closely with you, caring about your frustrations that only they can fix. Sadly there are still so many silos left to break up.
Most people with such a title are actually something like “Automation Engineers”, “Infrastructure Engineers”, “Operations Engineers”, “Site Reliability Engineers”, etc, that are involved in a DevOps “process”, “initiative”, “culture”, etc.
However it’s also misguided to assume that specialities don’t exist. You can have infrastructure guys, developers, security folk; and there will be overlap between each role but it’s impossible to be the master of each trade.
I agree that arrogance is an unpleasant trait but arrogance can take many forms, rudeness to colleagues, or over assuming ones own technical capabilities in adjacent fields.
b) In my experience, the application developer is held responsible for the application's behavior in production. In the luckiest .01% of scenarios, there might be an infrastructure engineer with appropriate permissions and free time trolling the Slack support channel at the moment you report the issue. Otherwise 99.99% of the time infrastructure will not acknowledge of investigate anything complicated or subtle with just one service owner complaining. The infrastructure group, organizationally, is graded on shipping new platform features and on coarse KPIs for the performance of the platform as a whole; nobody is getting paid to investigate the weird bugs of some application team somewhere.
No, it doesn't eliminate such distinctions. My view of DevOps is more about ensuring that automation is used as much as possible to meet objectives.
It's definitely not about making everyone a homogeneous developer unit that can work on every problem.
People come in all shapes and sizes, some are more competent with certain things than others, others have a lot more experience with certain things. That's aside from the whole preference thing - not everyone wants to or has an interest in managing infrastructure.
Maybe when you have a handful of developers and a small set of infrastructure, that's fine - but at a certain point you start to require more and more specialised knowledge. Yes, even when you're all-in on Cloud and using all the SAAS/PAAS products out there.
>Otherwise 99.99% of the time infrastructure will not acknowledge of investigate anything complicated or subtle with just one service owner complaining.
Yeah, that's an organisational problem from the sound of it.
I think it’s best to consider the original source which is this talk from Flickr: https://youtu.be/LdOe18KhtT4
("10+ Deploys Per Day: Dev and Ops Cooperation at Flickr").
Directly it talks about joining developers and operations into the same team- later Patrick Debois would refer to this as DevOps and a year later the first DevOps days in Ghent was organised (also by Debois).
We really need some new terminology.
I'm being a little flippant here though, I am aware that developers are often asked to wear the sysadmin hat too.
That is the responsibility of system administrators. Application developers have no business on a production machine. If your sysadmins don't have the technical skills to diagnose these problems, they are incompetent and must be replaced.
The actual result of this is that the sysadmins are not replaced, and the application developers end up in an emergency conference call at 3am to tell the sysadmins which buttons to click on the production environment, since they’re not allowed access themselves.
DevOps is not a role or role segregation, it’s about aligning incentives and outcomes across functions in an org (hopefully through collaboration, tooling, and knowledge transfer).
The caveat is that if your org is fundamentally broken, none of the above applies or works and it’s all lipstick on a pig.
This kind of access shouldn’t be your go-to but it’s important to have a battleshort.
But I would be lying if the typical logging setup was this simple. It’s basically all remixes of ELK and while they’re fantastic when they work there are a lot many moving parts that can individually fail.
Great! hand waving. What are you going to do is this log streaming process stops working, and cannot restart? Kill the machine and lose the logs? That might be ok for your webapp, it could be trouble for more serious applications.
> You’d be surprised at how much less stress there is when you can have alerts on log events from your application when things break instead of a support call or a support ticket.
Giving SSH access doesn't mean you don't have event logs.
> stopped teaching people to treat their app and server as a second home, as a pet, that must be nurtured
In our case, nobody is treating anything as a pet. Our servers are mostly spot/preemptibles, but we still have SSH access. Because it makes it really easy to debug if things do go badly wrong. From being able to see htop output, to being able to upload custom binaries to investigate an issue, or to smoke test a code change... it's a huge time saver.
Give me SSH or give me death.
The discussed sining flow probably works better with cloud infrastructure. Afaik it's one of the ways hashicorps vault can be used for SSH.
This occasionally pops up at reddit and the "key in ldap" way feels surprisingly unknown/uncommon. Many use ansible.. but it requires guaranteed cleanup of revoked keys....
To overcome this issue, you end up storing a set of public keys in the servers themselves, thereby going back to where you started.
https://access.redhat.com/documentation/en-us/red_hat_enterp...
Our systems people get local accounts on machines for disasters like LDAP-down. Everyone else is LDAP-only, including ssh keys.
This is nowhere close to "where we started". Now we have a handful of privileged accounts and centralized auth management across thousands of other accounts.
Centralized logging and centralized auth are pretty much mandatory above some size. Without them, you literally do not know who is doing what.
That way you get both emergency access in case LDAP is broken, as well as a way to make sure old personal access gets revoked after one month.
There are of course also DR considerations for that, as always, it is turtles all the way down.
SSH configs in distros are moving towards not even doing rDNS lookups (which generally just delay logins for the timeout period when there's a problem).
That doesn't mean some hybrid solution doesn't have merit though.
And in this blog entry I am summarizing what I learned and what I think is a very good approach now.
Thanks!
The lab I work at recently had a "worry" during recent attacks on UK science infrastructure that someone might have compromised an unknown private key someone might have stored on a third part server. The institutional response was to switch to password-based ssh only for a month (so lots of people used sshpass...) and then revoke - at the bastion - every single key listed in every single authorized_keys for every one of thousands of users in the system, before turning key access on again.
I've wondered why they didn't switch to an ssh certificate system. My best guess is the complication, maintenance, and risk profile of setting up a secure key-signing system to do this signing automatically - presumably this acts as a single point-of-failure for compromising the entire network.
The general response to those compromises has been a disaster area. Especially as (in the absence of any post mortem, as far as I know) the attack presumably involved the general password-spraying that was happening around then. Users were made to look responsible and suffer because systems had local privilege escalations due to poor management -- which, to be fair, might be due to relying on cluster vendors who "take security very seriously", as they say. That has been around the top of the threat list for decades too, it's just that it's now much larger scale. Then you got private keys purged that might be used to access systems on other sites with less risk than typing passwords on insecure systems -- arbitrarily deleting users' data.
It's not possible to have 100% of your keys on a short duration, so you have to cook up a revocation system.
That's not my idea of a fun time. Auditing keys on disk is much less trouble than properly managing CRLs.
We use a cronjob to regularly fetch, verify and deploy updated authorized_keys on servers.
I don’t know about you, but at least at my org, waiting for tens of thousands of machines to sync an authorized keys file when doing a ton of onboarding or a very very selective off-boarding... I’d rather minimize the amount of machines I touch and doing that via an authorized_keys push via ansible is untenable at scale or volume.
If a single file change is untenable at scale or volume, how does your organization manage version deployments, or even something small like an emergency reconfiguration, without collapsing or having to cope until "patch tuesday"? What you describe sounds untenable to me.
I can see this working nicely in relaxed companies that are comfortable waiting for the signed keys to expire (but those companies probably also won't like the inconvenience of expiring keys).
With that said, there is an advantage, in that a revoked key will eventually expire even if the cronjob breaks.
It’s pretty easy to audit a server for a key in a file or a key file on disk.
https://docs.ansible.com/ansible/latest/collections/ansible/...
It would be really nice to have a solution to this problem that doesn't require the user to go to first principles involving Alice, Bob, and Charlie.
If you’re on AWS also consider high value add built ins like EC2 instance connect or ssm session manager so you can manage host access via IAM.
[1] https://linux.die.net/man/5/sshd_config [2] https://news.ycombinator.com/item?id=24620842
Disclaimer: main developer and co-founder here.
We started theoapp [1] in mid 2018 and we introduced it at FOSDEM 2019 [2]. We then launched AuthKeys [3], a company that offers it as SaaS.
From a technical standpoint, we chose a different approach than that described by this thread's O.P. More specifically we implemented our service as a client/server suite to integrate with the standard sshd authorizedKeysCommand feature. Rationale and more technical details can be found at the links below.
We've been in "stealth mode" until now (this is our very first public post!), while running the service with selected customers since December 2019.
Recent developments will soon allow us to widen the product’s offer with a self-hosted solution of AuthKeys.
Feedback from the HN community is more than welcome: we'd like to hear whether the website is effective and clear enough on explaining how AuthKeys works [4]. We realized the topic comes with a certain degree of complexity for some potential users while others really want to quickly spot how the internals work and how we managed to make it secure to trust and easy to use.
We’ll be happy to answer any questions on the matter, and to offer promo codes to HN readers in case you wish to give it a go.
[1] https://github.com/theoapp/ [2] https://archive.fosdem.org/2019/schedule/event/theo_keys_man... [3] https://authkeys.io/ [4] https://authkeys.io/how-it-works
The same TGT can also be used for other neat things like secure and transparently automounted homes and other directories via NFS4 or CIFS.
You no longer have to deal with SSH keys whatsoever and all the management that goes with them: When users get their access revoked on AD, they get their SSH access revoked as well. You can have group based authorization (only those in the SRE group can access this class of QA endpoints), so when dozens of people a month are being added and removed from the various groups, you don't have to worry about giving them keys/access. They can SSO from their laptops, so all they have to do is open PuTTY and they can connect away without even typing their usernames and passwords. etc.
Lots of these new generation "devops" and "full-stack developers" haven't had the experience of AD and Kerberos, so they spend all this time, blog posts, money, etc. to reinvent the wheel.
Sad really.
But since Catalina there's now a great Kerberos SSO plugin that you can push through MDM. Previously this was known as enterprise connect but only available from Apple professional services.
if you have access to all your machines through public internet you're doing it wrong (ie stop solving a problem that you should not have in the first place).
The proper way to do this is to have a bastion host (or a jump host) where you strictly control access. Someone leaves you revoke their access. (extra: have the access be protected via 2FA that depends on the person having an active [LDAP/corp] account)
There is another layer to this where the access to the bastion is allowed only from the corporate network (ie you need to VPN into the corpnet to be able to access the jump host). You leave, you no longer can access the bastion.
the ultimate level to this is that you should not ssh willy nilly into your production hosts (to the degree that this should not even be possible). you should have a solution for pushing the logs + instrumentation (ie metrics) that makes it so that you don't need to do this in production.
If manual, I would even consider doing it every week with the caveat that it would be a large attack vector.
This way when a developer leaves the company, you just remove their entry in the LDAP DIT and all and every access is removed.
I haven’t used Nix personally but I’ve used Guix - I want to go further with it and I’m curious how you’re managing this.
Is that covered in the NixOS approach as well?
Sounds interesting!
Is the CA not a single point of failure also?
I agree with one users comment, ansible can solve all of this too...
However, the attack vector here is much lower given you can have this on a local machine which is not even connected to a network if you like.
The single point of failure for central management solutions is in having them as a running service - if that goes down, you are in trouble.
And that is far more likely than your CA becoming compromised somehow.
Also in practice, you can also run two or more CAs (we do that), so you can deprecate a full CA if the need arises.
What year is this? 2010? No, application logs shouldn't reside on servers, and Devs shouldn't need access to servers either.
It’s allows developers to manage their own certificates (adding a new machine or rotating a key) whilst allowing us to use the GCP IAM tooling to grant access to certain machines - all without hosting bastion servers ourselves or exposing the servers themselves to the public internet.
As it’s based on PAM its also been relatively painless to integrate with other functions like audit logging.
If you’re using GCP already - I’d highly recommend it!
TrustedUserCAKeys /etc/ssh/ca.pub
Or does this scenario either require the user juggling certs, or me generating certs containing all concurrent claims?
One thing I'm sure would work if the user has generated two separate public-private key pairs and you sign two different certs.
Not so sure about having several certs for the same key.
If I was in that situation, I would probably generate a new cert which contains the concurrent claims and is short-lived, but we also don't have extremely many different roles.
Also SSM has some annoying glitches like e.g. https://github.com/aws/amazon-ssm-agent/issues/274
Edit: found this blog post that shows how the two can work together https://skorfmann.com/blog/aws-ecs-instance-connect-meets-aw...
No, doesn't require managing ssh keys. SSM can work with SSH keys but it's not required. The downside I found is that all users connect with the same uid/gid which does not work great for bastion hosts where user profiles are required (e.g. connect to EKS API). You could work your way around this with documents permissions per user, but it's not great.
The user management is handled by IAM.