OpenSSH 10.6p2 has been released with a workaround for a critical vulnerability
hachyderm.io
hachyderm.io
Does anyone know why musl does this? Are they the only C library that behaves this way?
For a C library that famously tries to adhere to POSIX as much as possible, this seems like a crazy choice. From POSIX standard:
Upon successful completion, crypt() shall return a pointer to the encoded string. The first two bytes of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
https://pubs.opengroup.org/onlinepubs/9699919799/functions/c...
https://hachyderm.io/@ncopa@fosstodon.org/117564809789053485
Though I am surprised that all of the code examples do not use constant-time strcmp(). Actually, I wonder if even constant-time strncmp() would still leak information about whether the account is disabled because in that case both arguments are much shorter? Food for thought.
But yeah, the crypt(3) API is very 80s C, the man page even features a lengthy description of a long-deprecated and utterly broken hashing algorithm using DES that is even implied to be broken in the text itself (and the warning message is so old it refers to "parallel computers" as though they were a new concept). Given that the default mode is to use a broken hashing algorithm, I'm shocked anyone actually uses the libc crypt(3) -- surely it's trivial to just write your own that isn't awful...?
In any case, the fact you need access to an existing hash to not select something stupid like DES is all due to crypt(3) being a wacky API and while callers could work around it by selecting more reasonable defaults it's a pretty big foot gun.
Error handling is underspecified. The spec doesn't give you enough well-defined behaviour, meaning everyone lists pages of error-handling extensions they have added.
Pedantic, but more "author's understanding of the spec" which isn't always correct.
> assume nonstandard behavior
I go back and forth on this one (spec vs standard behavior). It feels like at some point the most widely used impl -is- the spec, as we've seen in other places (web standards).
> Upon successful completion, crypt() shall return a pointer to the hashed password [...]. Otherwise, it shall return a null pointer and set errno to indicate the error.
> The crypt() function shall fail if:
> [ENOSYS]
> The functionality is not supported on this implementation.
For example on Debian the manpage says:
> Upon error, crypt_r, crypt_rn, and crypt_ra write an _invalid_ hash to the output field of their _data_ argument, and crypt writes an invalid hash to its static storage area. This string will be shorter than 13 characters, will begin with a ‘*’, and will not compare equal to _setting_.
To the extent that systems that are configured in the vulnerable manner exist in the wild, I’m surprised they weren’t exploited years ago by trivial ssh bots.
P.S. Wow, I’m reminded of just how awful traditional UNIX-y C APIs and code are.
P.P.S. The musl code in question is nuts. The API it implements is awful. Seriously, here’s the manpage:
https://man.archlinux.org/man/crypt.3.en
You can’t call this function correctly. sshd’s “fix” is basically to assume that a too-short return is an error return instead of a success return.
PAM is also awful, but this is worse.
"It starts with a bold claim: "ssh accounts that should not accept a password will accept any password between 256 and 1024 characters". But then it keeps adding qualifiers:
- sshd compiled with musl
- not using PAM
- password logins enabled
- at least one account with '*' as the password "hash"
- at least one OTHER account with a valid password
The conditions are all ANDed, miss a single one and you're fine."
This doesn't seem to be nearly as big as an issue as they're making it out to be
Disabling PAM has been "generally good security advice" for decades; I remember suggestions to not use it in the 2000s.
Password logins are often enabled, especially on multi-user and default setups.
If alpine uses the * feature, it's going to be pretty common. Gentoo has it, as does Ubuntu.
If password logins are enabled, you're almost guaranteed to have at least one user with a password (root on all my boxes has one, even if I never use it and it's random garbage).
So the cheese lines up pretty well, this affects quite a bit.
If password logins are enabled, someone almost certainly has a password.
I'm seeing people I generally trust in the scene memeing on twitter, while others are freaking out. Its hard for me to tell how fucked we are, I guess
I think this time around it's mainly Alpine Linux and a few custom-rollers (Gentoo with musl could easily get to this place).
I think this will likely become standard. Nevertheless, a few stormy months are ahead.
PasswordAuthentication=no
at this point the _last_ thing i care about is my email.
where am i? am i ok? how are my friends/loved ones?
once those are in place, it becomes clearer how to get back to "normal"
On systems that receive regular updates, SSH can fail for many reasons other than authentication problems, after all.
Is it fast? No, but that's a feature.
But it does mean I get to lean on the existing processes and identity verification already part of the government bureaucracy and the courts as my backdoor.
Locally: Same, but printed out, laminated, in a safe place with other documents.
Who can presumably get you in touch with the relevant embassy or consulate or, failing that, local authorities.
And this is assuming the server is somewhere else on the cloud.
Or I have proper backups synced across the world on different providers. The likelihood of everything failing is not a vector I worry about. If that happens I have bigger problems to worry about.
Openssh allows: password auth with weak passwords, PAM with potentially insecure PAM modules, gssappi, configurable ciphers, etc. Wireguard has none of those.
Each of those auth methods adds more possible vulnerable code paths and accidentally insecure configurations. It's very reasonable to believe that wireguard auth, since it has far fewer moving parts and configuration knobs, is likely to have fewer ways it can be insecure.
crypt() returning `*` is insane especially since these kinds of sentinels are all over /etc/passwd
I suppose if people started writing lots of code that crashed on error in Rust some enterprising developer might improve the function to always return a non-error value and set it to ‘*’ in error cases.
One is to return NULL for errors.
Another is to make the string result a parameter and use a return code to signal error.
A third, less common but still out there, is an error parameter.
Failure to observe these is on the caller. Forcing the caller to consider the error is probably where rust shines.
But a good C programmer is thinking about these idioms when they encounter a function signature for the first time, or they write one.
h = file_open(path)
data = file_read(h)
file_close(h)
return data
return file_open(path).success(h -> h, file_read(h)).success(h, data -> file_close(h); data)
This is the root of a lot of the evil. Yes, humans want to program for the happy path. It is a lot more fun. However, that bias leads to so much lack of diligence to properly deal with the unhappy path.
You ideally want to attack the problem at two ends:
- make the syntax noise-free so you can see what is happening
- build primitives at the syscall and stdlib level for common footguns
You would still need to deal with domain-specific logic errors, but you have at least reduced the surface area exposed to casual attacks.
It is not. Being extremely restrictive in the happy path, throwing all over the place, but have generic error handling code at the top level (e.g. HTTP entry point) makes for some pretty robust, stable, easy to read software. Albeit it typically has somewhat rough user experience.
The thing is, that's a pretty good place to start. From there on, you can handle exceptions one by one, surgically, and make things nicer.
I do prefer Maybe/Option/Either style returns in some cases though. Makes for a cleaner API.
So each method has its uses.
Which is why exceptions in Java can bubble up the stack? You don't have to catch it in a particular method you can let it go to the caller of the method, and so on.
Rust makes handling and mapping `Result` locally more ergonomic, even though `?` makes propagation easy too. Rust, however, also makes it easy to lose the context during propagation and end up with useless `Error`.
Right idea at the wrong time.
Having been on the receiving end of debugging that a few times, I no longer mind the in-band error handling, although admittedly I'm inexperienced enough to be confident its better. Just for the random code I've had to read where I just need to know exactly how something I didn't write and will never read again works, and whether the error is handled and how, i love reading in-band error handling, even if I too dislike the way an error laden routine must be written. Over time I've come to appreciate elegance "in the large" as opposed to the code I see in any individual function.
I have written very little Java, though, so this is more of an armchair analyst position.
The same way that layered code should react/wrap/translate when it comes to return types, it should do the same for thrown exceptions, and checked-exceptions helps you do that. They're both results of computation.
Another Rust self-suck session rendered completely irrelevant. Again.
I'm not sure this gets the point across when (according to https://sshhole.com/) it effectively means "any other user with a real password"
"If you cannot update yet, at least disable password logins: set PasswordAuthentication no in your sshd configuration and restart sshd. Either one closes the hole." (only AI would characterize a critical SSH implementation vulnerability as a hole closing game)
An attacker who can reach your SSH port can log in without knowing any password. It takes one connection, works every time, and needs no retries. (It sounds like the writer is bragging about the exploit instead of describing it. AI does this)
The attacker needs the name of an account whose password field is * (an account with no password set, which is normal for key-only accounts), and needs the machine to have at least one other account with a $5$ or $6$ password hash. The attacker then sends a very long password (257 to 1024 bytes). One user with a password is enough: it makes every account without one reachable. (same as above made stronger with the 'statement: statement2' format)
"Users coming from a directory service through musl-nscd (nslcd, winbind, sssd), since they have no shadow entry." (a human would be more clear about what a shadow entry is and give more context, I think)
Hmm the rest of the article (the remaining 70-80%) seems quite human.
Anyway, not sure why I spent 10+ minutes on this. It just jumps at me right away.
As a grad student I read and graded a lot of student philosophy papers (well before LLMs), and I just remember the exhaustion of it. It made me appreciate how much I could otherwise just read something, get information or enjoyment out of it, and not focus on each sentence construction.
Just to say, I do believe you, I get that if you have to put up with it so much in general, it must prime you to find it everywhere. You have to read so much more now, I'm sure, so you're trained to be in that mode. It makes sense! I am just sorry its like this for you and most everyone else now. It sounds exhausting.
Then it was a surprisingly customer service intensive process to get the keys from the customers. I never did manage to automate it fully. Weird enterprise systems I have never heard of with strange key formats. ssh1 keys, email mangling them. Their internal processes requiring a password.
And way to many "I am sorry but that was the private key you sent me, That key is now compromised and you will need to remove it and generate new keys anywhere it was used"
I almost regret not allowing passwords, almost, despite the headaches key based is still much better.
Still safter in general, but not bulletproof.
[of course not having the credentials to leak does not protect us from this sort of bug, it is just one factor in a multipart security stance]
Current state (now we have been bought and are effectively more than an order of magnitude bigger): I definitely don't have opportunity to make a unilateral decision like that. Even though the larger company has a more “fair enough, feel free to go elsewhere” attitude when a small client says “if you can support × that is a dealbreaker for us” then we could previously afford to take as our stance, some of the bigger clients are definitely paying us enough, and do use passwords, that someone above me would overrule a keys-only decision.
You must present a key first after which you must enter a password.
Isn't this safer than "PasswordAuthentication no", even if it's a different setting?
Funnily enough, they don’t recommend checking pubkey auth is enabled and working. By blindly (or agenticly) following those instructions the risk is getting locked out of some machine.
Seems quite reasonable. It’s quite obvious that black hats have day jobs as White hats and have been privy to embargoes for probably decades.
Anything longer is putting systems at danger. I can recompile the package with a patch (even if Ubuntu/RHEL takes 3 weeks to do it) myself in a pinch, and I'd like to have a chance for it before the bug gets re-discovered by malicious party
So not a ssh issue but an issue specific to Linux using "musl libc without PAM".
They instead pass in a different user’s hash (to avoid timing attacks) but end up comparing the result from musl against the actual user account, defeating musl’s simplistic safeguard to avoid this very issue.
I’d probably still call this a musl thing. Errors should result in a NULL return even if the application crashes from there, but at the very least a return value intended to not match any actual hash in /etc/shadow should be a random string or predefined UUID or something other than *.
Never understood why Alpine is used to actually run anything in a container - if you statically link binary, then you don't need alpine as a base and if you don't what is the point? Don't @ me about image size, even without layer caching we're talking about tiny difference that is often dwarfed by size of your runtime.
Musl itself is a great example of taking "simple and lightweight" so seriously that everyone else gets to deal with the complexity. Slow allocator, underoptimized memcpy/string routines, limited locale support, no NSS, and endless glibc compatibility headaches.
But hey, at least libc is small. %%Proceeds to pull in megabytes of <any other>malloc, and other libraries to replace the slow parts of musl.%%
And now sshhole.com is a particularly spectacular example of what happens when "simple" meets the rest of the software ecosystem.
There's a good chance similar flaws will be discovered elsewhere.
One occurrence is not a statistic and should not be used as evidence - it's just an example of a vulnerability that due to various bad circumstances lead to a particularity bad outcome. This logic is in my opinion equally flawed as calling alpine safe now because this particular vulnerability has been fixed.
I'm not a big fan of blaming and I think it's just a result of bad circumstances, but it's interesting to see how some seem to blame musl, but one could argue further that musl documented the behavior of their API which was ignored by openssh and so on
With dynamic linking, you just upgrade the library on your system and you're done. With static linking, every binary that uses the library has to be rebuilt.
Looking at the unpatched Ubuntu things in my life, and I'm pretty close to asking Claude to update things to see how quickly Alpine can get close, assuming responsiveness can stay this fast and quality.
Not every C vuln is a memory safety problem.
Now, the decision to return `"*"` instead of letting poorly written code crash on `NULL` is not good either.
Given how easy it is to just forward that error to the caller, I think most Rust developers would just choose to do that.
That's exactly what you get with Result and Option, it's impossible to forget.
Compare:
char* hash = crypt(key, salt);
// We crash here on the hash derefence, it's no clear from the code
// that we were supposed to check for NULL or that we forgot.
if (!strcmp(hash, expected)) {
}
let result: Result<Vec<u8>, Error> = crypt(key, salt);
// We MUST write this line, which is a very clear signal
// that we can panic here.
// This can be grepped for or forbidden.
let hash: String = result.unwrap();
// if result == expected { would type error
if hash == expected {
}You are really uninformed: different programming languages have very different volumes of bugs per line of code
do you think bugs/loc is the same between brainfuck and rust?
no the study failed to show any difference in bug count for a specific set of common used languages, where C and Python were both part of the dataset.
I don't think programming languages have a big impact on bugs per line when every line is written by a human. Now given that I was talking about bug counts, not severity.
kind of reminds me how the "research" suggests that there are no practical differences between dynamically typed and statically typed languages...except we know from practical experience that there is, and the literature there is horribly flawed.
we have real-world evidence that rust leads to significantly less bugs in the real world
https://blog.google/security/rust-in-android-move-fast-fix-t...
It showed that there's no big difference. From the link:
> Combining corrections for all of these aforementioned items, the reanalysis revealed that only 4 out of the original 11 languages correlated with abnormal defect rates, and even for those the effect size is exceedingly small.
--
> we have real-world evidence that rust leads to significantly less bugs in the real world
no, we only have evidence for memory bugs, that's not what I was talking about
In Rust, the type of the return value would be a `Result<T, E>`, which means the chain of events that led to this would never have started in the first place.
Edit: at the beginning, crypt always returned a value. So hypothetical Rust return type would be a String, and then it would be retained for backward compatibility. Or something else would happen, this is a complex logic bug triggered by a 25 year old chain of events.
That's what led some people to write code where it crashes without checks. Which is still their mistake. But if the type was something that can't be implicitly used like an always-present value, that would have prevented the situation, even with those same people.
Trying to change a function return type from e.g. `string` to `string?` would make the backwards-incompatible breaking change immediately visible, and break the builds of all code that relied on not having to check it back when it was `string` - at compile time.
But it turns out one can have a compile-time error forthis in C as well: https://godbolt.org/z/8E9z66cfs
Easier to formulate invariants and check/prove them. Better language correlates with safer software.
You are right, C with manual memory management, no lifetime management, lack of phat pointers and a culture of leet has problems with deranged maddening bugs that other languages don't have.
But this bug first shows how inband signaling is bad news usually. But worse it's a failure to adhere to the spec properly. If you implement a library and you don't adhere to the spec you're going to bone someone.
Perhaps even worse is obviously the spec doesn't come with any standard testing framework. That was also the root of Apples infamous goto bug.
oh right, because c is a crusty old language with barely any features.
let hash = match result {
OK(s) => s,
Err(s) => s,
};
@my-huge-pony does make a good point elsewhere in the thread: if the potential error was not foreseen and the code was written originally to return `String` instead of `Result<String, Error>` then even Rust would be in a pickle.
what will it take to move away from c? older devs retiring. Planck said that "science advances one funeral at a time" and i fear it will be the same for software.
Rewriting it to Rust is fine attempt, just hard to call it is unix/posix anymore. It may not matter but semantically true.
I'll take forking and porting, thank goodness for LLMs.
in theory could someone do it? yeah, just like in theory someone could program a calculator to randomly rm -rf ~ when pressing +.
but in practice would anyone do it? no.
At the bottom of that it says dropbear is unaffected.
> on debian, ubuntu, fedora or anything else with glibc? going through pam? dropbear? openbsd? you're fine
Except in the case where their images are built with OpenSSH instead of Dropbear, which is the case for some manufacturers. (Turris, was noted as one such vendor)
https://gitlab.alpinelinux.org/alpine/aports/-/work_items/16...
It didn't give me the impression that I was dealing with a secure piece of software, it seemed like they were dealing with very complex issues while laughably failing at the industry basics that were settled years ago.
Similarly Ubuntu was also the only other distro that fucked this simple permissions config up, and lately with the latency to fix the TCP vuln, it seems to confirm that these are not reliable distros for the coming security era.
> This release contains a critical security workaround that affects Linux systems that use the musl libc and PAM disabled.
But that there's nothing about it on their security page.
Even if it's not technically a bug in their code, the fact that they've made a "critical" release to address it suggests that it should probably be listed on the security page as well. But maybe (hopefully) they just haven't updated the security page yet.
> Upon successful completion, crypt() shall return a pointer to the hashed password; the first two bytes of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
Musl returns '!' or '*', not null.
“Only two remote holes in the default install, in a heck of a long time!”
I'm pretty sure the misleading wording is deliberate, OpenSSH is an excellent but non-commercial project which a bunch of entities would love to gain control of, in order to bug it with their spyware - thus the constant nagging on the bot-infested web.
The OP should be flagged.