Ask HN: What do you use for SSH key management of teams? | Hacker News Reader

Ask HN: What do you use for SSH key management of teams?

31 pointsmpaepper32 comments

Everyone uses ssh keys to manage access to their servers, but so far I haven't found a great (ideally open) solution to manage those keys.

Manual management is tedious and error prone.

In particular, I want to be able to add and remove keys and assign user's access rights to certain servers.

If I remove a key, the access to all servers should be revoked.

What do you use for this?

View on HN

If you're willing to entrench yourself deeper into the AWS ecosystem you can go completely keyless and manage access solely through IAM by inventorying your instances in AWS Systems Manager - then you can start SSH sessions right from the AWS CLI itself[1].

[1] https://docs.aws.amazon.com/systems-manager/latest/userguide...

This still requires an SSH key.

From the link:

To start a session using SSH, run the following command:

ssh -i /path/my-key-pair.pem username@instance-id

No, the aws start-session command does not require a keypair - you're reading further down in the instructions about other ways to connect.

Indeed, thanks for the correction.

Okta (scale-ft) at work, evaluated gravitational teleport in the past.

In the past (company with <1000 employees), I set up nss-cache and a saltstack system on a timer to regularly deploy new keys from LDAP (we used bastion hosts to control for any dangers of config drift and I wanted zero SPOFs during steady state). I would say this is the least likely to fail under all scenarios and is therefore the best choice unless it is somehow untenable (large number of employees, or extremely dynamic user creation/deletion)

I haven't used any of these myself, but you might want to take a look at Facebook's approach to the problem[0]. It's rather different and innovative.

You might also find Smallstep [1] and SSH Lockbox [2] interesting.

[0] https://engineering.fb.com/security/scalable-and-secure-acce...

[1] https://smallstep.com/blog/diy-single-sign-on-for-ssh/

[2] https://github.com/half-cambodian-hacker-man/ssh-lockbox

Are you on AWS? If not, probably skip this.

The other answer about AWS Systems Manager is good. I recommend it.

Other way is piggybacked off of AWS IAM and CodeDeploy[0]. Users load their personal keys into CodeDeploy and you manage them through IAM. Every container/SSH machine syncs keys from CodeDeploy every 10 minutes (whatever you set the cron to).

Lastly, you can connect EC2 Instance Connect[1]

[0] https://github.com/widdix/aws-ec2-ssh

[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Connect-...

I use SaltStack. I run the state setting the keys fairly often and enforce the presence of only the keys I set. This also prevents people from having the impression that they can add their own.

I think I remember reading somewhere that there's a way of using LDAP / ActiveDirectory (I'm pretty much the only guy running Linux on my machine) but I haven't looked into it yet.

we use signed certs with authorized principals to manage access [1] and sign the certs after successful MFA. if you need a non-interactive connection you can use token authentication to fetch a cert.

[1] https://engineering.fb.com/security/scalable-and-secure-acce...

Signed certs for ssh is IMHO the best solution for managing this problem in larger orgs. Nice to see Facebook published their process around it.

https://gravitational.com/blog/how-to-ssh-properly/

Signing certificates for hosts and users. Never deal with authorized_keys files ever again.

Seconding that. We use Hashicorp Vault as the SSH CA that issues keys certificates.

> Never deal with authorized_keys files ever again.

For servers with an access to multiple users via SSH with PKI auth you have to use AuthorizedPrincipalsFile anyway. edit: and to avoid gazillion issuing CAs for managing group access.

https://gravitational.com/teleport

Teleport (https://gravitational.com/teleport/) has been fairly useful, basically a ssh bastion solution with short term (hours) keys. They have an open source tier. Downside is that the company is really small, support is scarce after the check clears, and all their future development seems geared towards kuberneetes and IOT. So if you just want a reliable bastion with key management they aren’t really interested in your demographic.

We centralize everything in openldap and developed a simple django app to centrally manage them.With a combination of sudo-ldap and a custom schema (OpenSSH-LPK) and using the openssh configuration AuthorizedKeysCommand we manage authentication and all sudo permissions.

We use the Keybase SSH CA. It's secure, removes the need for individual ssh keys, a d moves things over to chatops.

https://keybase.io/blog/keybase-ssh-ca

how do you manage the other parts of their identity? LDAP is a common choice..

AFAIK a SSH CA can be a good solution for this.

See e.g. https://news.ycombinator.com/item?id=16615307 for some more info.

Standard PKI infrastructure can be used as a reasonable way to manage SSH access. Can issue signed certificates and revoke them as per usual.

We simply put all the team's keys in an S3 bucket. Each server regularly syncs the bucket and updates the authorized_keys file.

Host inventories modelled with ansible. For user X to gain access to ansible host group Y, you just need to make a merge request with your key and host group. CI syncs ansible configs with the hosts.

There is also a cron that checks authorized_users vs git and sends email when something is out of sync.

In the past I've used Hashicorp Vault to handle this kind of situation. Granted, it's an additional piece of infrastructure to manage but Vault has been pretty solid for this kind of situation and others where you need to safely manage secrets.

If you dont mind, how did you set it up? I have vault right now but I dont know exactly how to use it for ssh'ing.

The process is pretty simple but their documentation is pretty good. When I was starting out I found this video which helped me get started: https://www.hashicorp.com/resources/manage-ssh-with-hashicor...

Couldn’t you just use whatever you already use to manage your infrastructure? They are just files on a server so you would manage them with a deploy just like anything else. Initial keys can be added through userdata at instance launch.

You can store the keys in a database, ldap or whatever and set the AuthorizedKeysCommand of your ssh server to a command that looks up the keys given the user. The keys store can be the same as your user database (eg: AD etc)

I use jumpcloud (https://jumpcloud.com). Works well, free for small teams. Not open.

I would strongly suggest you evaluate KMS. While not open, it has the support and development behind it to make it secure. This is going to get you IAM control of your users and allow group roles.

It also integrates with AWS well, and of course your own applications.

"AWS Key Management Service (KMS) makes it easy for you to create and manage cryptographic keys and control their use across a wide range of AWS services and in your applications."

https://aws.amazon.com/kms/

What does KMS have to do with SSH key management?

KMS is part of secure managed solution: "The secrets in Secrets Managers are encrypted with AWS Key Management System (KMS), and every version of the secret is encrypted with a unique data encryption key." [1]

[1]: https://aws.amazon.com/blogs/security/how-to-use-aws-secrets...

Whatever configuration management you already use.

Reply on news.ycombinator.com