Show HN: Securely set the same password on thousands of Linux servers | Hacker News Reader

Show HN: Securely set the same password on thousands of Linux servers

github.com

23 pointsseletskiy47 comments

I'd argue that if you're setting the same password on thousands of servers, what you're doing is inherently insecure.

Seriously, I can't imagine this scenario in real life without ssh-keys.

shadowd is not against SSH keys in any way. It can serve them as well.

The main point is always having password access for cruicial services, so any given ops engineer can login from almost everywhere. In real life there can be different situations, anecdotal, but at some point of time I have to fix problem via SSH shell from my mobile phone.

It depends. You can have many development servers/nodes, which are infrastructurally "inside" the company and is not accessible from outside. They should be managed somehow, security requirements can be considerably less than at production cluster, and as an example it's always good to have rescue password for the ops team that can be used to login to the server in case of it's unresponsible in any other way.

Also, shadowd do not make assumption about same password distribution, it can handle any amount of passwords by your choice, depending on severity and importance of service.

I think Google is on the right track with assuming that there is no perimeter and it will only be a matter of time before every company of any decent size/budget will follow suit (maybe within the next 3-7 years). The network security architecture that can best be described as a hard shell with a squishy inside has been proven time and again to be dangerously full of assumptions.

I guess I'm not clear on why randomizing hashes for identical passwords across thousands of servers is enough of a problem that I would run a new HTTP service to handle it, given that SSH keypairs largely solve this problem already.

If you mean that the problem is that they have the same salt, then just call "passwd" individually on each instead of copying /etc/shadow.

But anyway, just disable passwords and login with SSH public key authentication.

Calling passwd on each host (imagine hundred of them for tens of users) and manually setting same password is not the way to go. The idea is to have same password will be setted up on numerous amount of nodes without storing or transferring it.

SSH keys are good, but there are always should be rescue password which can be entered via IPMI or by physically attaching to the server. Also, `sudo` password (which is also handled by /etc/shadow) is a good measure on production servers and should be set somehow. shadowd resolves it.

>rescue password which can be entered via IPMI or by physically attaching to the server.

If it's an emergency you can reboot into a single user/recovery/rescue state and reset it. No need to increase the attack surface just for that.

>Also, `sudo` password (which is also handled by /etc/shadow) is a good measure on production servers and should be set somehow.

I agree about a sudo password being a good idea, but if we're talking about the "right" way I'd say a authentication directory backend of some sort would be better suited for that.

Often, the only way to know what's wrong with server/software is to login through IPMI and see it while it's still alive. Reboot will lost that information. Don't see exactly how enabling root password login at physical/IMPI console will increase attack surface (not SSH).

Regarding authentication backend I've already tried to address that vision. Unable to become root when auth backend offline/unreachable/misconfigured is significantly worse than having local shadow which is always works.

The chance of your central authentication system being down at the same time you have to troubleshoot an issue with a live system seems unlikely. And for those rare cases you can just enable the account in single user and troubleshoot after the reboot.

> manually setting same password is not the way to go

Agreed, if you have something like dish installed, something like

dish "echo root:password | chpasswd"

should do the trick to everything dish knows about.

This is a simplification, but the general idea isn't too far off. If you don't know why its a simplification, then doing it this exact way is probably a bad idea.

I'll call it oversimplification. It very error-prone and very difficult to automate. Of course, you can always tell your ops engineer to run something like `xargs -n1 -I{} ssh {} echo ... \| chpasswd < hosts.list` manually, but we are talking about automatisation of routine tasks, not "solving" them by shell scripts.

I get that there's a very specific use case for this, but it doesn't really need all the extra features like a REST client/server and key distribution. Most people have already built their infrastructure and config management and would maybe like a tool that just generates X different salted hashes of the same password and then copies them to remote sites. (Which commonly you'd implement as a single expect tcl script)

I prefer well written, well working, well documented and packaged software that solve problems, not tcl scripts or any other shell scripts that is very difficult to maintain.

shadowd/shadowc is exactly the tool that generates X different salted hashes and distributes it to remote sites, as you described. It embeds to any infrastructure almost painlessly.

It isn't explained in the README; but why is this approach better than a tried, tested and secure LDAP setup with SSH Keys?

I understand LDAP can be quite obtuse to configure (particularly after OpenLDAP dropped config files), but once you get it working it is rock solid.

Completely agreed. Synchronized passwords are risky compared to LDAP.

LDAP is definitely proven (esp in a LAN/non-cloud environment), but isn't perfect: if LDAP is DoS'ed (even from the very servers that it's authenticating for), then you can't log into any servers (or if a maintenance window exceeds the cache window, or even if it's just taken offline via hostname change or something). It also doesn't help much with user-based UX (i.e., key rotation, granting someone additional rights usually has to go through a centralized IT dept, etc) and caching is pretty awful, and it's a pain to securely configure. Lots of risk of inadvertently opening a hole through an insecure PAM/NSS configuration or ACL's.

The worst part with LDAP is the long and sordid history of LDAP and KRB5 root-level vulnerabilities.

Even with all of these drawbacks, LDAP probably less risky than storing (even different) hashes of the same password across thousands of boxen!

Disclaimer: I'm the CTO @ https://Userify.com ; we build an SSH Key management tool that's specifically designed for cloud servers.

Fundamentally you need a password in a sealed envelope in a safe in the ops area, so when LDAP is in a hopeless situation, a field tech can be talked over the phone into logging in as root using the sealed envelope to access a new LDAP server, or something.

Also its "easy" to set up insecure ldap or more precisely its easy to not deny insecure ldap connections, but its hard to mess up something like Kerberos to be overly permissive... But it doesn't matter, it just changes the scenario to "kerberos is dead for whatever reason, need root login to fix".

Thanks for valuable note about README, it's certanly needed to be explained.

But, I hope, I describe point about LDAP and stuff in replies to other discussion members comments.

Why is this solution preferable to making a hash preimage at least three orders of magnitude more difficult to calculate, e.g., using SHA512 instead of SHA256?

I don't see the point actually. If one figures out the password from the hash its going to work on all servers anyway...

I guess attacker can find a collision to hash (using bruteforce), not real password (because real password should be very strong)

As asdfaoeu noted above, the chances of this actually happening are very close to zero for any password anyone would realistically use.

Nope. Attacker can figure out "some" password (e.g. will find collision for specific salt+hash), that will be valid only for the server it already has access too, so it's not the big deal.

All other nodes will have different salt+hash pairs, so brute-forced password will be not valid for them.

I must be missing something, then: if you take input password A and salt/hash it individually for each server, and the attacker finds the valid input password A via brute force on one, why wouldn't it work on all of them? They aren't brute-forcing to find the salt or the hash, they're brute forcing input password A.

The attacker isn't going to find a collision that is not your password unless you are using a stupidly weak hash algorithm. The only input the attacker is likely to find is the original password and this is assuming it has less than 128bits of entropy.

That'd only apply if they're using very long, random passwords. For a 128-bit hash that'd be at least ~20 chars using the entire keyboard, and over 22 if alphanumeric only. Your docs mention SHA256/512, so double or quad those.

If your attacker is able to "figure out" 128-bit+ passwords, then you've probably got other problems. And if the point of using passwords is for emergency logins from phones, I'm not sure how useful a 80 char pass is. Might as well keep the ssh key saved somewhere and type it in eh?

I stopped using passwords on servers. That's right, I don't know my password on most of them. How? I use my ssh key pair to authenticate. At one point I regocnized that there are much worse things than a compromised server. The attacker can easily replace /bin/login with a program that does in fact log you in, but also emails them your password silently. Then they can try all your other hosts at their leisure and exploit your access until you figure out what's up.

Because of this I now use pam_ssh_agent_auth and try to never enter my actual password into any remote server. Of course this still means there is a risk: I have to forward my ssh agent to the remote server, and an attacker with root access there may in fact hijack that connection and connect to other servers using my credentials. However, (1) this can only happen so long as I am connected to the server, (2) I can be notified every time my private key is used to respond to a challenge, and (3) the attacker never learns any of my personal secret values (password or private ssh key).

Lol there is a technology called LDAP, avoiding storing passwords anywhere on the server.

Don't go and reinvent the wheel if there are far better technologies widely adopted!

The rules for "Show HN's" aren't the same as for the rest of the site, and a negative comment staring with "Lol" probably wouldn't clear the civility bar even for a normal thread.

https://news.ycombinator.com/showhn.html

I'm not certain that I'd agree with you about this comment. Which rule does it break? The poster believes this technology already exists and started a relevant discussion about the pros and cons of the existing implementations. I don't think you could consider "Lol" so offensive that the post should disappear.

You simply shouldn't combine negativity and dismissiveness in a Show HN comment. You probably should avoid dismissiveness altogether, but this particular mixture is especially toxic here.

I didn't find it offensive, but it did come across as childish and ignorant, and therefore I stopped reading at that first word. Opening a discussion by laughing in the face of the other person (or group of people in this case) reeks of ignorance, even if the person doing so is actually an expert.

If the poster wants to be taken seriously, that's his or her choice. But however ignorant and childish it is, it's still something someone could say "in a face-to-face conversation" as per the rules.

I'm not arguing this was a nice thing to say, I'm arguing it shouldn't disappear. We have the power to downvote people when they say things that don't fit in the community but pass a very basic civility guideline. Comments that aren't substantive should disappear. Comments potentially with some meat but offensively presented should just be downvoted, in my opinion.

This is normally the case on HN, but it isn't the case on "Show HN's". The special "Show HN" guidelines were introduced by 'dang in a thread about a year ago; you should track that thread down and read the whole thing for context, or just take my word for it. :)

I don't generally want to be the HN rules police, but this particular rule is a very good one, which took the site some time to be adopted, and I think it's worth speaking up for.

> I'm not arguing this was a nice thing to say, I'm arguing it shouldn't disappear.

Note that I didn't call for it to be censored or even modded down, I said I stopped reading it because I felt it was a childish response to a serious issue. It seems you somehow inferred that I felt it should "disappear", but I don't feel that way. I simply choose not to read the rest of it, which hurts no one's feelings.

The author put a lot of work into this (I assume). "Lol" and then dismissing their work isn't very nice. You should ask yourself if this is the type of comment you would say in person (because the author is likely reading these comments).

Perhaps a more productive comment would ask "what advantages this has over LDAP"

That is not a really wise strategy. LDAP is fine for general user accounts when you need to give access to employees on some shared hosts. For production hosts, you often don't want a SPOF such as LDAP for accounts. And I promise you this, the day your LDAP or network will go down, if you don't have at least a single local user to ssh with, you'll regret it.

While I agree with no reinventing the wheel unless there's good reason to do so, LDAP is a bloated, over complicated piece of $#!£¥.

I'm not saying that LDAP is simple, but compared to jenkins, its a delight to setup (although openLDAP is just stupid, why keep the config inside the LDAP instance it's self(and yes I know the justification....))

Agreed, dealing with LDAP would be far, far easier if there were a single monolithic config file (or hell, even a folder full of config files) instead of the unbelievably complex LDIF process.

LDAP, or LDAP + Radius (for integration with a second factor) is still a dynamite technology and pretty much required once a network / team reaches a certain size. There's definitely demand for an "EasyLDAP" or the like that offers sane defaults and easy config for deployment in a cloud environment.

You can use the config file option - it's just not the default anymore in openldap.

Also, having config in the db means you can make online configuration changes that take effect instantly and are replicated to all other replicas.

LDAP is a remote login provider, works through PAM-module and is nowhere near "lightweight" as it supposed to be. Also, it stores only one hash for original password, not many. If you do not have cached version of auth info on the server and remote login provider is offline, you have big problems.

I can't imagine a single reason why I will do remote login service for the root user on the front-end load balancing server of a high load project.

shadowd is not remote login provider, it only serves /etc/shadow hashes for local and possibly offline use.

Sorry but no, this is just not the right way to do it.

with SSSD there is no excuse for not using AD/LDAP.

Having done both, shipping /etc/passwd and using AD, its far more effective, secure, and crucially stable to have LDAP/AD as the single source of truth.

Its not like its difficult anymore, you can even buy in AD from AWS. (just don't try and do machine accounts, you're better off using puppet/ansible for that sort of thing.)

Think about it, you wouldn't ship /etc/hosts anymore would you? why do the same with user accounts.

I guess, you didn't address any single issue I mentioned. I don't see, why it's far more effective, secure and stable to have proposed stack as the single source of truth. Given the example, that we do not have LDAP stack already integrated into dev/ops process, it's seems like an overblow for me to integrate it from scratch because of it's massive configuration, maintenance and ugly design.

The right way to do for me is simplicity and using tools, that offers solution for small problems; using overcomplicated bloatware doesn't seems to be right way to do, especially now, when we have that rapid growth in software development.

I would ship /etc/hosts if there will be major performance impact of doing millions of DNS queries across network under high load. It depends on the concrete case.

SSSD cache for as long as you need. If you don't have an auth server or network then you are in deeper trouble. (even AD can be n-tier active active)

AD/LDAP allows SSO for everything ssh, bash, gmail, AWS console, etc. Because its a centralised, when someone leaves, or their account is compromised then its one place to change it. Crucially its one source of truth across the entire estate, not just for "servers" or "desktops"

As for "massive configuration, maintenance and ugly design." Its really not. Also purity of design is not getting the job done. Your approach is ok if you only have a few servers and a handful of users.

The point of ansible/puppet/salt/$other playbooks is that 90% of the work is done for you. All you have to do is input your users, and roll out the SSSD conf (after testing) Even by hand openLDAP isn't that hard.

As for the rhetorical question about /etc/hosts you wouldn't because thats what nslcd/sssd/Program caches are for.

I think a LDAP/LDAP+Kerberos backend is more secure for user accounts but I'd agree that the root password doesn't belong in it. But personally I think disabling the root account (like OS X or Ubuntu) makes the most security sense. You can always get into the account in single user mode.

Reply on news.ycombinator.com