musl’s crypt() returns the string * as an error indicator when the key is over 256 bytes long. * is also exactly what the stored field contains for every account with no password, so the “hash” matches and the login succeeds.
Does anyone know why musl does this? Are they the only C library that behaves this way?
For a C library that famously tries to adhere to POSIX as much as possible, this seems like a crazy choice. From POSIX standard:
Upon successful completion, crypt() shall return a pointer to the encoded string. The first two bytes of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
Thanks yeah just woke up, finished making my way through the single-article website and still unclear if other libcs do the same thing (the article says libxcrypt picked it up, but only after musl did it).
Apparently because some devs didn't check for NULL which resulted in crashes. Weird choice to fix other peoples' bad programming by deviating from the spec.
To be fair (as the sshole site mentions) it is also necessary to fake a password hash check to avoid timing side channels revealing to remote clients whether users exist, so returning a sentinel value is not a horrifically unreasonable solution to make sure even naive code does approximately the right thing.
Though I am surprised that all of the code examples do not use constant-time strcmp(). Actually, I wonder if even constant-time strncmp() would still leak information about whether the account is disabled because in that case both arguments are much shorter? Food for thought.
But yeah, the crypt(3) API is very 80s C, the man page even features a lengthy description of a long-deprecated and utterly broken hashing algorithm using DES that is even implied to be broken in the text itself (and the warning message is so old it refers to "parallel computers" as though they were a new concept). Given that the default mode is to use a broken hashing algorithm, I'm shocked anyone actually uses the libc crypt(3) -- surely it's trivial to just write your own that isn't awful...?
The spec only defines a single error state - ENOSYS, that is basically only for systems that don't implement crypt at all.
Error handling is underspecified. The spec doesn't give you enough well-defined behaviour, meaning everyone lists pages of error-handling extensions they have added.
This is either a bad engineering choice or an intentional bug. An empty string would have been better. Why '*' instead of, say, a tilde, @ symbol, or something that generally isn't used?
Wow, what a bug. I expect the exploit is literally as simple as trying to ssh in as a user with the password has ‘*’ in /etc/shadow and typing anything as the password.
To the extent that systems that are configured in the vulnerable manner exist in the wild, I’m surprised they weren’t exploited years ago by trivial ssh bots.
P.S. Wow, I’m reminded of just how awful traditional UNIX-y C APIs and code are.
P.P.S. The musl code in question is nuts. The API it implements is awful. Seriously, here’s the manpage:
"It starts with a bold claim: "ssh accounts that should not accept a password will accept any password between 256 and 1024 characters". But then it keeps adding qualifiers:
- sshd compiled with musl
- not using PAM
- password logins enabled
- at least one account with '*' as the password "hash"
- at least one OTHER account with a valid password
The conditions are all ANDed, miss a single one and you're fine."
This doesn't seem to be nearly as big as an issue as they're making it out to be
Disabling PAM has been "generally good security advice" for decades; I remember suggestions to not use it in the 2000s.
Password logins are often enabled, especially on multi-user and default setups.
If alpine uses the * feature, it's going to be pretty common. Gentoo has it, as does Ubuntu.
If password logins are enabled, you're almost guaranteed to have at least one user with a password (root on all my boxes has one, even if I never use it and it's random garbage).
So the cheese lines up pretty well, this affects quite a bit.
If password logins are enabled, someone almost certainly has a password.
No, no don't let all of that distract you. We gotta hype this. This is the new SSH2SHELL dude. Have we registered the domain yet? I'm scheduled on Fox News tomorrow.
> We are making this release as fast as possible (we learned of the problem yesterday) because we have recently observed multiple cases of rapid bug rediscovery by AI agents, sometimes within hours, and so now consider long embargo periods to be a dangerous practice that primarily benefits attackers.
I think this will likely become standard. Nevertheless, a few stormy months are ahead.
How do you break back into your digital life? You wake up in a strange place. Something happened. Everything's gone. No wallet/keys/smartphone/tablet/laptop. How would you get back into your digital life given only a laptop with access to the Internet, unless something, somewhere is passwordauthenticate=yes.
PasswordAuthentication=no usually means that ssh stops directly reading /etc/passwd to authenticate, not that it stops accepting passwords completely. You'll still be able to log in with a correct password, provided KbdInteractiveAuthentication is still on, PAM is available, and the PAM policy allows for it.
The email at the root of my identity is tied to a country code domain in my country. There are robust legal processes for enforcing my status as the registrant and regaining control of it. Once I have access to that, I can use that to gain access to everything downstream.
Is it fast? No, but that's a feature.
But it does mean I get to lean on the existing processes and identity verification already part of the government bureaucracy and the courts as my backdoor.
Remotely: An encrypted file with a seed / master passwords to all the other data, published under a public url you can remember, but not linked anywhere and not listed.
Locally: Same, but printed out, laminated, in a safe place with other documents.
I walk to my pc and type ssh-keygen. If i can’t do that I’ve got bigger problems. “I must be able to recover remotely using a memorised credential” is a terrible requirement.
Yeah your vault has password authentication. Your password manager in other words. Your vault is where you store your private ssh keys, and or the passwords for the keys.
Another thing I would recommend is having your servers in a wireguard net and only allowing ssh from inside that virtual network. There is no need to publicly expose ssh.
I was always uncomfortable that it's on by default and surprised anyone uses it it. It's also more cumbersome compared to just using a key. I feel like using keys is the password manager analogue for SSH.
Is this actually effective against Alpine? The default install seems to use ! as the placeholder in /etc/shadow instead of * and trying the attack doesn't seem to work.
IMO this is where languages like Rust shine. The design patterns built into the language for handling errors (Result) - force you to handle errors / function results with sentinels, null pointers, or errnos are so much more error prone than handling a Result<T,E> using a match statement.
crypt() returning `*` is insane especially since these kinds of sentinels are all over /etc/passwd
I guess this is where the culture around rust shines. You can write crypt that returns a string in rust just fine, just like you can write crypt that returns a tagged union in C. Cultural differences and syntactic sugar change which version is idiomatic.
I suppose if people started writing lots of code that crashed on error in Rust some enterprising developer might improve the function to always return a non-error value and set it to ‘*’ in error cases.
The culture is downstream of the design IMO. In general, people write the API that is obvious. In C, returning a string is obvious, while defining a custom OptionalString type is high friction. In Rust, returning Option<String> is as easy and obvious as returning a String.
All the standard Rust libs use results. The standard C libs use errno, return 0, return -1, return EOF, return nullptr, segfault, or of course UB. Other libs dream up things like returning "*". C++ has exceptions in some areas but not others, and if you turn them off, they're segfaults. I actually like C, at least more than C++, but the error handling is the worst part.
Sure, you could write deliberately bad code in Rust by unwrapping everything everywhere, but the point is that it is a distinct choice to do so. You can’t forget to handle errors - you have to make the choice to disregard them. If you forget to check for NULL in C it won’t complain and it’s dependent upon you to remember to do so.
In the vast majority of programs, errors and non-results are irrelevant at the call site. You want to program for the happy path and not wrap every single call with error checking code or `try/catch/finally`.
h = file_open(path)
data = file_read(h)
file_close(h)
return data
is the code you want to read. This essentially is:
return file_open(path).success(h -> h, file_read(h)).success(h, data -> file_close(h); data)
A language with thoughtful error handling primitives will support such code.
The question is _where_ the error needs to be handled. It's almost never directly at the call site. And then there's a few issues with checked exceptions and Java language things (RMI, closures, etc) which make checked exceptions cumbersome. An exception (checked or not) might be worse than sum types, but they're still a thousand times better then returning an error in-band.
Better design breeds better habits. In Java, you can end up handling a DNS exception 20 layers up the stack, even if your call never actually triggers a DNS lookup, simply because some method deep in the call stack declares it as possible. The culture is to propagate or wrap exceptions rather than translate them into domain-specific errors near the source.
Rust makes handling and mapping `Result` locally more ergonomic, even though `?` makes propagation easy too. Rust, however, also makes it easy to lose the context during propagation and end up with useless `Error`.
I agree but I think was also because Java didn't have sum types (or still doesn't? years I haven't read Java) and rethrowing is super easy, while Rust kinda nudges you into the right direction (if you need to rethrow at all, think if you want to rethrow verbatim or wrap with a semantic error type).
I always thought the ideas was great, but type system was holding it back. If you could have expressed that your thrown exceptions are the same as some other method's (e.g. a method of a class given as a parameter), and some other classes, then it could have been a lot less annoying. Powerful, even! But Java did take its sweet time to get even type parameters.
I have written very little Java, though, so this is more of an armchair analyst position.
Amen, people saw the symptoms of "needs more syntactic sugar" and misinterpreted it into "rigorous error handling is bad".
The same way that layered code should react/wrap/translate when it comes to return types, it should do the same for thrown exceptions, and checked-exceptions helps you do that. They're both results of computation.
Unfortunately this is also an area where a Rust programmer would likely just reach for the platform libc, because you really want to be compatible with exactly how the platform is hashing passwords. Once you know about the inconsistency, you can make predictable wrappers around it, but platform legacy is gonna keep throwing these curveballs.
Not really related to the topic, but AI-popular sentences and language structure stand out to people right away these days right? I noticed some.
"If you cannot update yet, at least disable password logins: set PasswordAuthentication no in your sshd configuration and restart sshd. Either one closes the hole." (only AI would characterize a critical SSH implementation vulnerability as a hole closing game)
An attacker who can reach your SSH port can log in without knowing any password. It takes one connection, works every time, and needs no retries. (It sounds like the writer is bragging about the exploit instead of describing it. AI does this)
The attacker needs the name of an account whose password field is * (an account with no password set, which is normal for key-only accounts), and needs the machine to have at least one other account with a $5$ or $6$ password hash. The attacker then sends a very long password (257 to 1024 bytes). One user with a password is enough: it makes every account without one reachable. (same as above made stronger with the 'statement: statement2' format)
"Users coming from a directory service through musl-nscd (nslcd, winbind, sssd), since they have no shadow entry." (a human would be more clear about what a shadow entry is and give more context, I think)
Hmm the rest of the article (the remaining 70-80%) seems quite human.
Anyway, not sure why I spent 10+ minutes on this. It just jumps at me right away.
An aside but one nice thing I realize about not using LLMs is, I guess, I am not tortured by this kind of thing, because presumably I am not seeing it all the time. I can't even imagine quite being in the mode of reading this kind of text so closely like this, really. Its a security advisory, I don't really have my critical prose-reviewer hat on.
As a grad student I read and graded a lot of student philosophy papers (well before LLMs), and I just remember the exhaustion of it. It made me appreciate how much I could otherwise just read something, get information or enjoyment out of it, and not focus on each sentence construction.
Just to say, I do believe you, I get that if you have to put up with it so much in general, it must prime you to find it everywhere. You have to read so much more now, I'm sure, so you're trained to be in that mode. It makes sense! I am just sorry its like this for you and most everyone else now. It sounds exhausting.
> because we have recently observed multiple cases of
rapid bug rediscovery by AI agents, sometimes within hours, and so now
consider long embargo periods to be a dangerous practice that primarily
benefits attackers.
Seems quite reasonable. It’s quite obvious that black hats have day jobs as White hats and have been privy to embargoes for probably decades.
Embargo should last at most to the moment fix is made and code passes test suite.
Anything longer is putting systems at danger. I can recompile the package with a patch (even if Ubuntu/RHEL takes 3 weeks to do it) myself in a pinch, and I'd like to have a chance for it before the bug gets re-discovered by malicious party
What if the test suite passes in the middle of the night? Should all administrators interrupt their sleep and get up to recompile their servers immediately?
We have clients (regulated banks and insurers of various sizes) that send us feeds via SFTP. You might be surprised what proportion of them ignore our “we strongly recommend key based authentication” and ask for password auth instead.
When I set up a client sftp drop I had the luxury of saying "key based access only"
Then it was a surprisingly customer service intensive process to get the keys from the customers. I never did manage to automate it fully. Weird enterprise systems I have never heard of with strange key formats. ssh1 keys, email mangling them. Their internal processes requiring a password.
And way to many "I am sorry but that was the private key you sent me, That key is now compromised and you will need to remove it and generate new keys anywhere it was used"
I almost regret not allowing passwords, almost, despite the headaches key based is still much better.
Don't most cloud providers default to that? I know I had to force some to deploy with a key and even then PasswordAuth was often on and had to be disabled.
I went and checked this. I can admin my server with Cockpit using either the web service (using my username/password) or the desktop client (via private key) without issue even with SSH password authentication disabled. I do usually disable the web service so had to reconfigure to check.
Well, OpenSSH contributes to this by mixing hashes from the shadow file. The musl code would be correct if OpenSSH passed in the hash from the user account that is being logged-in.
They instead pass in a different user’s hash (to avoid timing attacks) but end up comparing the result from musl against the actual user account, defeating musl’s simplistic safeguard to avoid this very issue.
I’d probably still call this a musl thing. Errors should result in a NULL return even if the application crashes from there, but at the very least a return value intended to not match any actual hash in /etc/shadow should be a random string or predefined UUID or something other than *.
The potential affected space is huge.. it does not matter if most docker images ship without openssh.. it does not matter that most security conscious image builders do not enable plain passwordauth.. it does not matter that ssh port is not commonly exposed.. the amount of alpine docker images in current live use over the whole internet is such a vastly huge number that you're going to bet that even given a small percentage of those 'misusing' docker and their containers being affected is still going to result in a a huge number of affected containers.
i think it's just unbelievable there isn't a bigger push to move away from c to languages that can prevent stuff like this from happening. this and just the pure unsafety. it's 2026. it's like pure negligence at this point, the equivalent of building a bridge with matchsticks because you like using it more.
How would using a language different than C help here? At it's core this bug is a logic bug, where "*" == crypt(x), because crypt in musl started returning "*" in some cases some years ago. This could be Rust. This could be Go.
Another comment said that returning a * was to avoid returning null which would crash users of the library who didn't check for null. That seems like a C smell to me
>because crypt in musl started returning "*" in some cases some years ago
In Rust, the type of the return value would be a `Result<T, E>`, which means the chain of events that led to this would never have started in the first place.
I write firmware where the program has to run for years unattended.
You are right, C with manual memory management, no lifetime management, lack of phat pointers and a culture of leet has problems with deranged maddening bugs that other languages don't have.
But this bug first shows how inband signaling is bad news usually. But worse it's a failure to adhere to the spec properly. If you implement a library and you don't adhere to the spec you're going to bone someone.
Perhaps even worse is obviously the spec doesn't come with any standard testing framework. That was also the root of Apples infamous goto bug.
Rust would not save you here as this is a logic bug. Nor would, well, really any other language. I suppose someone might claim that Rust's result type would save you but it really wouldn't: I can easily make the error type a String or pretty much any other type with little consequence.
Rust would absolutely have saved you here. Even with `Result<String, String>`, which is pretty poor design, there's no confusing `Ok(s)` with `Err(s)`. Sure you can go out of your way to write something like
let hash = match result {
OK(s) => s,
Err(s) => s,
};
but at that point I feel like the library is forgiven (except for the poor error design).
@my-huge-pony does make a good point elsewhere in the thread: if the potential error was not foreseen and the code was written originally to return `String` instead of `Result<String, Error>` then even Rust would be in a pickle.
it's unbelievable to me that there isn't a larger movement of developers to move away from c towards better & safer languages. look at how much pushback the rust4linux folks get, primarily from c developers who are simply afraid of change. imagine engineers not adopting safer building practices because "they prefer the old way"....it's disgusting honestly.
what will it take to move away from c? older devs retiring. Planck said that "science advances one funeral at a time" and i fear it will be the same for software.
Cars without seatbelts and airbags chopping off driver's heads in relatively slow collisions? Well guess what, that could happen in a modern car as well if the driver sticks their head out the window while going through a very tight tunnel!
> set PasswordAuthentication no in your sshd configuration and restart sshd.
Funnily enough, they don’t recommend checking pubkey auth is enabled and working. By blindly (or agenticly) following those instructions the risk is getting locked out of some machine.
I agree entirely, but I'm not sure changing `crypt` to return `"*"` instead of `NULL` on failure qualifies. It's much more of a considered decision, which happens to be very wrong.
Except in the case where their images are built with OpenSSH instead of Dropbear, which is the case for some manufacturers. (Turris, was noted as one such vendor)
It didn't give me the impression that I was dealing with a secure piece of software, it seemed like they were dealing with very complex issues while laughably failing at the industry basics that were settled years ago.
Similarly Ubuntu was also the only other distro that fucked this simple permissions config up, and lately with the latency to fix the TCP vuln, it seems to confirm that these are not reliable distros for the coming security era.
both alpine and musl are great in my opinion, and as far as I know theres no empirical evidence that it is more or less secure overall than alternatives. would you like to elaborate your opinion?
IMO only good thing about alpine is that it's a good environment to build stuff that links musl statically.
Never understood why Alpine is used to actually run anything in a container - if you statically link binary, then you don't need alpine as a base and if you don't what is the point? Don't @ me about image size, even without layer caching we're talking about tiny difference that is often dwarfed by size of your runtime.
Musl itself is a great example of taking "simple and lightweight" so seriously that everyone else gets to deal with the complexity. Slow allocator, underoptimized memcpy/string routines, limited locale support, no NSS, and endless glibc compatibility headaches.
But hey, at least libc is small. %%Proceeds to pull in megabytes of <any other>malloc, and other libraries to replace the slow parts of musl.%%
And now sshhole.com is a particularly spectacular example of what happens when "simple" meets the rest of the software ecosystem.
Static linking makes distribution easier, but it's an absolute security nightmare.
With dynamic linking, you just upgrade the library on your system and you're done. With static linking, every binary that uses the library has to be rebuilt.
It's nice to see how quickly this is being addressed, and likely monitored for any unintended side effects.
Looking at the unpatched Ubuntu things in my life, and I'm pretty close to asking Claude to update things to see how quickly Alpine can get close, assuming responsiveness can stay this fast and quality.
Seems slightly confusing that the OpenSSH release notes page says:
> This release contains a critical security workaround that affects Linux systems that use the musl libc and PAM disabled.
But that there's nothing about it on their security page.
Even if it's not technically a bug in their code, the fact that they've made a "critical" release to address it suggests that it should probably be listed on the security page as well. But maybe (hopefully) they just haven't updated the security page yet.
musl libc violates the POSIX documented behavior of `crypt()`¹
> Upon successful completion, crypt() shall return a pointer to the hashed password; the first two bytes of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
OpenBSD is not affected, so apparently they are doing something right. But also glibc linux is not affected and pam linux is not affected, so it is only musl linux with the remote hole.
You became a victim of a very misleading post title. This isn't an OpenSSH problem at all, it's a musl problem.
I'm pretty sure the misleading wording is deliberate, OpenSSH is an excellent but non-commercial project which a bunch of entities would love to gain control of, in order to bug it with their spyware - thus the constant nagging on the bot-infested web.
musl’s crypt() returns the string * as an error indicator when the key is over 256 bytes long. * is also exactly what the stored field contains for every account with no password, so the “hash” matches and the login succeeds.
Does anyone know why musl does this? Are they the only C library that behaves this way?
For a C library that famously tries to adhere to POSIX as much as possible, this seems like a crazy choice. From POSIX standard:
Upon successful completion, crypt() shall return a pointer to the encoded string. The first two bytes of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
https://pubs.opengroup.org/onlinepubs/9699919799/functions/c...
There's a very detailed trace of how the bug was developed over decades at sshhole.com and the relevant discussion in https://news.ycombinator.com/item?id=50378228
Thanks yeah just woke up, finished making my way through the single-article website and still unclear if other libcs do the same thing (the article says libxcrypt picked it up, but only after musl did it).
1 reply →
Apparently because some devs didn't check for NULL which resulted in crashes. Weird choice to fix other peoples' bad programming by deviating from the spec.
To be fair (as the sshole site mentions) it is also necessary to fake a password hash check to avoid timing side channels revealing to remote clients whether users exist, so returning a sentinel value is not a horrifically unreasonable solution to make sure even naive code does approximately the right thing.
Though I am surprised that all of the code examples do not use constant-time strcmp(). Actually, I wonder if even constant-time strncmp() would still leak information about whether the account is disabled because in that case both arguments are much shorter? Food for thought.
But yeah, the crypt(3) API is very 80s C, the man page even features a lengthy description of a long-deprecated and utterly broken hashing algorithm using DES that is even implied to be broken in the text itself (and the warning message is so old it refers to "parallel computers" as though they were a new concept). Given that the default mode is to use a broken hashing algorithm, I'm shocked anyone actually uses the libc crypt(3) -- surely it's trivial to just write your own that isn't awful...?
4 replies →
TBH that sounds like a straight up bug in that it deviates dangerously from the spec.
The spec only defines a single error state - ENOSYS, that is basically only for systems that don't implement crypt at all.
Error handling is underspecified. The spec doesn't give you enough well-defined behaviour, meaning everyone lists pages of error-handling extensions they have added.
Which is notable because musl is more known for adhering to the spec to the point where they break programs that assume nonstandard behavior.
5 replies →
This is either a bad engineering choice or an intentional bug. An empty string would have been better. Why '*' instead of, say, a tilde, @ symbol, or something that generally isn't used?
I'm not entirely sure an empty string, tilde, or @ symbol would really have been better, except in retrospect.
Wow, what a bug. I expect the exploit is literally as simple as trying to ssh in as a user with the password has ‘*’ in /etc/shadow and typing anything as the password.
To the extent that systems that are configured in the vulnerable manner exist in the wild, I’m surprised they weren’t exploited years ago by trivial ssh bots.
P.S. Wow, I’m reminded of just how awful traditional UNIX-y C APIs and code are.
P.P.S. The musl code in question is nuts. The API it implements is awful. Seriously, here’s the manpage:
https://man.archlinux.org/man/crypt.3.en
You can’t call this function correctly. sshd’s “fix” is basically to assume that a too-short return is an error return instead of a success return.
PAM is also awful, but this is worse.
Maybe I'm misunderstanding, but u/tonnydourado pointed out on another thread https://news.ycombinator.com/item?id=50378228#50380029
"It starts with a bold claim: "ssh accounts that should not accept a password will accept any password between 256 and 1024 characters". But then it keeps adding qualifiers:
- sshd compiled with musl
- not using PAM
- password logins enabled
- at least one account with '*' as the password "hash"
- at least one OTHER account with a valid password
The conditions are all ANDed, miss a single one and you're fine."
This doesn't seem to be nearly as big as an issue as they're making it out to be
Alpine matches the first.
Disabling PAM has been "generally good security advice" for decades; I remember suggestions to not use it in the 2000s.
Password logins are often enabled, especially on multi-user and default setups.
If alpine uses the * feature, it's going to be pretty common. Gentoo has it, as does Ubuntu.
If password logins are enabled, you're almost guaranteed to have at least one user with a password (root on all my boxes has one, even if I never use it and it's random garbage).
So the cheese lines up pretty well, this affects quite a bit.
If password logins are enabled, someone almost certainly has a password.
4 replies →
No, no don't let all of that distract you. We gotta hype this. This is the new SSH2SHELL dude. Have we registered the domain yet? I'm scheduled on Fox News tomorrow.
1 reply →
FWIW, I made a bit of an error in my very quick analysis. See: https://news.ycombinator.com/item?id=50380581
> We are making this release as fast as possible (we learned of the problem yesterday) because we have recently observed multiple cases of rapid bug rediscovery by AI agents, sometimes within hours, and so now consider long embargo periods to be a dangerous practice that primarily benefits attackers.
I think this will likely become standard. Nevertheless, a few stormy months are ahead.
musl only, the quick fix:
Which is a good advice in general.
How do you break back into your digital life? You wake up in a strange place. Something happened. Everything's gone. No wallet/keys/smartphone/tablet/laptop. How would you get back into your digital life given only a laptop with access to the Internet, unless something, somewhere is passwordauthenticate=yes.
> You wake up in a strange place. Something happened. Everything's gone. No wallet/keys/smartphone/tablet/laptop.
at this point the _last_ thing i care about is my email.
where am i? am i ok? how are my friends/loved ones?
once those are in place, it becomes clearer how to get back to "normal"
2 replies →
The somewhere for password authentication is a place you have physical access. Not ssh.
2 replies →
PasswordAuthentication=no usually means that ssh stops directly reading /etc/passwd to authenticate, not that it stops accepting passwords completely. You'll still be able to log in with a correct password, provided KbdInteractiveAuthentication is still on, PAM is available, and the PAM policy allows for it.
The email at the root of my identity is tied to a country code domain in my country. There are robust legal processes for enforcing my status as the registrant and regaining control of it. Once I have access to that, I can use that to gain access to everything downstream.
Is it fast? No, but that's a feature.
But it does mean I get to lean on the existing processes and identity verification already part of the government bureaucracy and the courts as my backdoor.
A scenario where all the backups are gone and the server is somehow the only thing that survived is not a very plausible one.
2 replies →
Remotely: An encrypted file with a seed / master passwords to all the other data, published under a public url you can remember, but not linked anywhere and not listed.
Locally: Same, but printed out, laminated, in a safe place with other documents.
I walk to my pc and type ssh-keygen. If i can’t do that I’ve got bigger problems. “I must be able to recover remotely using a memorised credential” is a terrible requirement.
3 replies →
Does this happen to you a lot? You may have other issues
Yeah your vault has password authentication. Your password manager in other words. Your vault is where you store your private ssh keys, and or the passwords for the keys.
I log in via console, either physical or virtual (the latter requires MFA though)
Sometimes you just have to start over. It’s not the end of the world.
1Password subscription and the memorized key. 1Password is the only place my ssh private keys are stored.
2 replies →
Another thing I would recommend is having your servers in a wireguard net and only allowing ssh from inside that virtual network. There is no need to publicly expose ssh.
Why is wireguards auth better than openssh auth?
6 replies →
I was always uncomfortable that it's on by default and surprised anyone uses it it. It's also more cumbersome compared to just using a key. I feel like using keys is the password manager analogue for SSH.
imo this is a good advice only if the devices that connect to the server have their home encrypted and ssh private key protected.
It's irrelevant. There's no reason this should be any more effective in the face of software bugs then anything else.
Is this actually effective against Alpine? The default install seems to use ! as the placeholder in /etc/shadow instead of * and trying the attack doesn't seem to work.
IMO this is where languages like Rust shine. The design patterns built into the language for handling errors (Result) - force you to handle errors / function results with sentinels, null pointers, or errnos are so much more error prone than handling a Result<T,E> using a match statement.
crypt() returning `*` is insane especially since these kinds of sentinels are all over /etc/passwd
I guess this is where the culture around rust shines. You can write crypt that returns a string in rust just fine, just like you can write crypt that returns a tagged union in C. Cultural differences and syntactic sugar change which version is idiomatic.
I suppose if people started writing lots of code that crashed on error in Rust some enterprising developer might improve the function to always return a non-error value and set it to ‘*’ in error cases.
The culture is downstream of the design IMO. In general, people write the API that is obvious. In C, returning a string is obvious, while defining a custom OptionalString type is high friction. In Rust, returning Option<String> is as easy and obvious as returning a String.
1 reply →
All the standard Rust libs use results. The standard C libs use errno, return 0, return -1, return EOF, return nullptr, segfault, or of course UB. Other libs dream up things like returning "*". C++ has exceptions in some areas but not others, and if you turn them off, they're segfaults. I actually like C, at least more than C++, but the error handling is the worst part.
Sure, you could write deliberately bad code in Rust by unwrapping everything everywhere, but the point is that it is a distinct choice to do so. You can’t forget to handle errors - you have to make the choice to disregard them. If you forget to check for NULL in C it won’t complain and it’s dependent upon you to remember to do so.
1 reply →
always has amazed me that there was so much hate for java checked exceptions where handling errors ought to be the norm.
In the vast majority of programs, errors and non-results are irrelevant at the call site. You want to program for the happy path and not wrap every single call with error checking code or `try/catch/finally`.
is the code you want to read. This essentially is:
A language with thoughtful error handling primitives will support such code.
13 replies →
The question is _where_ the error needs to be handled. It's almost never directly at the call site. And then there's a few issues with checked exceptions and Java language things (RMI, closures, etc) which make checked exceptions cumbersome. An exception (checked or not) might be worse than sum types, but they're still a thousand times better then returning an error in-band.
5 replies →
Better design breeds better habits. In Java, you can end up handling a DNS exception 20 layers up the stack, even if your call never actually triggers a DNS lookup, simply because some method deep in the call stack declares it as possible. The culture is to propagate or wrap exceptions rather than translate them into domain-specific errors near the source.
Rust makes handling and mapping `Result` locally more ergonomic, even though `?` makes propagation easy too. Rust, however, also makes it easy to lose the context during propagation and end up with useless `Error`.
I agree but I think was also because Java didn't have sum types (or still doesn't? years I haven't read Java) and rethrowing is super easy, while Rust kinda nudges you into the right direction (if you need to rethrow at all, think if you want to rethrow verbatim or wrap with a semantic error type).
Right idea at the wrong time.
1 reply →
They were overused, but some actually make sense, such as the IOException and SQLException
1 reply →
I always thought the ideas was great, but type system was holding it back. If you could have expressed that your thrown exceptions are the same as some other method's (e.g. a method of a class given as a parameter), and some other classes, then it could have been a lot less annoying. Powerful, even! But Java did take its sweet time to get even type parameters.
I have written very little Java, though, so this is more of an armchair analyst position.
Amen, people saw the symptoms of "needs more syntactic sugar" and misinterpreted it into "rigorous error handling is bad".
The same way that layered code should react/wrap/translate when it comes to return types, it should do the same for thrown exceptions, and checked-exceptions helps you do that. They're both results of computation.
Unfortunately this is also an area where a Rust programmer would likely just reach for the platform libc, because you really want to be compatible with exactly how the platform is hashing passwords. Once you know about the inconsistency, you can make predictable wrappers around it, but platform legacy is gonna keep throwing these curveballs.
It seems it used to return NULL, but they changed it for "convenience"
How is this more convenient, being able to display it as text?
Yeah too bad your adult-diaper language can't even host itself on most of the architectures where musl is relevant.
Another Rust self-suck session rendered completely irrelevant. Again.
> At least one local account has a $5$ or $6$ password hash.
I'm not sure this gets the point across when (according to https://sshhole.com/) it effectively means "any other user with a real password"
Not really related to the topic, but AI-popular sentences and language structure stand out to people right away these days right? I noticed some.
"If you cannot update yet, at least disable password logins: set PasswordAuthentication no in your sshd configuration and restart sshd. Either one closes the hole." (only AI would characterize a critical SSH implementation vulnerability as a hole closing game)
An attacker who can reach your SSH port can log in without knowing any password. It takes one connection, works every time, and needs no retries. (It sounds like the writer is bragging about the exploit instead of describing it. AI does this)
The attacker needs the name of an account whose password field is * (an account with no password set, which is normal for key-only accounts), and needs the machine to have at least one other account with a $5$ or $6$ password hash. The attacker then sends a very long password (257 to 1024 bytes). One user with a password is enough: it makes every account without one reachable. (same as above made stronger with the 'statement: statement2' format)
"Users coming from a directory service through musl-nscd (nslcd, winbind, sssd), since they have no shadow entry." (a human would be more clear about what a shadow entry is and give more context, I think)
Hmm the rest of the article (the remaining 70-80%) seems quite human.
Anyway, not sure why I spent 10+ minutes on this. It just jumps at me right away.
Agreed, I think all the tells you identified are correct. It is really off-putting to me still.
An aside but one nice thing I realize about not using LLMs is, I guess, I am not tortured by this kind of thing, because presumably I am not seeing it all the time. I can't even imagine quite being in the mode of reading this kind of text so closely like this, really. Its a security advisory, I don't really have my critical prose-reviewer hat on.
As a grad student I read and graded a lot of student philosophy papers (well before LLMs), and I just remember the exhaustion of it. It made me appreciate how much I could otherwise just read something, get information or enjoyment out of it, and not focus on each sentence construction.
Just to say, I do believe you, I get that if you have to put up with it so much in general, it must prime you to find it everywhere. You have to read so much more now, I'm sure, so you're trained to be in that mode. It makes sense! I am just sorry its like this for you and most everyone else now. It sounds exhausting.
Thank you for your comment, it's very refreshing (and comforting for me actually). Have a good day.
> because we have recently observed multiple cases of rapid bug rediscovery by AI agents, sometimes within hours, and so now consider long embargo periods to be a dangerous practice that primarily benefits attackers.
Seems quite reasonable. It’s quite obvious that black hats have day jobs as White hats and have been privy to embargoes for probably decades.
Embargo should last at most to the moment fix is made and code passes test suite.
Anything longer is putting systems at danger. I can recompile the package with a patch (even if Ubuntu/RHEL takes 3 weeks to do it) myself in a pinch, and I'd like to have a chance for it before the bug gets re-discovered by malicious party
What if the test suite passes in the middle of the night? Should all administrators interrupt their sleep and get up to recompile their servers immediately?
5 replies →
I think this is a long held belief of the project and why they have been excluded from bug embargo’s in the past.
Given Alpine’s broad use in the Docker ecosystem, this is going to hit a lot of folks, I think.
People generally don't expose SSH from within a Docker container I think.
no, ssh INTO a running alpine container. who runs sshd in a container when you can just attach a terminal?
1 reply →
Considering how much people whined about the MinIO images I would expect that it is not that uncommon.
Eh, most folks aren't hosting ssh servers with alpine containers and password auth enabled.
Who does PasswordAuthentication=yes in this day and age anyway.
We have clients (regulated banks and insurers of various sizes) that send us feeds via SFTP. You might be surprised what proportion of them ignore our “we strongly recommend key based authentication” and ask for password auth instead.
When I set up a client sftp drop I had the luxury of saying "key based access only"
Then it was a surprisingly customer service intensive process to get the keys from the customers. I never did manage to automate it fully. Weird enterprise systems I have never heard of with strange key formats. ssh1 keys, email mangling them. Their internal processes requiring a password.
And way to many "I am sorry but that was the private key you sent me, That key is now compromised and you will need to remove it and generate new keys anywhere it was used"
I almost regret not allowing passwords, almost, despite the headaches key based is still much better.
2 replies →
If you refused, would you get overruled internally?
1 reply →
it's the default for most systems. I can't remember a time I had to not turn it off.
Don't most cloud providers default to that? I know I had to force some to deploy with a key and even then PasswordAuth was often on and had to be disabled.
If you run a cockpit service on your server you will need password authentication enabled, unfortunately.
I went and checked this. I can admin my server with Cockpit using either the web service (using my username/password) or the desktop client (via private key) without issue even with SSH password authentication disabled. I do usually disable the web service so had to reconfigure to check.
As this is apparently affecting openwrt, probably a fair number of openwrt users, if I had to guess
I thought OpenWrt used Dropbear by default?
People who use AuthenticationMethods "publickey,password"
You must present a key first after which you must enter a password.
Just do no password at all.
1 reply →
>We consider the root cause of this bug to be outside OpenSSH itself, but we're in a good position to deliver a mitigation quickly.
So not a ssh issue but an issue specific to Linux using "musl libc without PAM".
Well, OpenSSH contributes to this by mixing hashes from the shadow file. The musl code would be correct if OpenSSH passed in the hash from the user account that is being logged-in.
They instead pass in a different user’s hash (to avoid timing attacks) but end up comparing the result from musl against the actual user account, defeating musl’s simplistic safeguard to avoid this very issue.
I’d probably still call this a musl thing. Errors should result in a NULL return even if the application crashes from there, but at the very least a return value intended to not match any actual hash in /etc/shadow should be a random string or predefined UUID or something other than *.
Randomly using a different users's hash seems so wrong to me, but maybe I've just been too deep in PII stuff for too long.
I'm sure some people running lightweight containers just had a heart attack.
I would imagine the vast majority of lightweight containers don't contain an ssh server?
You'd be surprised by the amount of people using containers that don't know/care what is inside.
The potential affected space is huge.. it does not matter if most docker images ship without openssh.. it does not matter that most security conscious image builders do not enable plain passwordauth.. it does not matter that ssh port is not commonly exposed.. the amount of alpine docker images in current live use over the whole internet is such a vastly huge number that you're going to bet that even given a small percentage of those 'misusing' docker and their containers being affected is still going to result in a a huge number of affected containers.
i think it's just unbelievable there isn't a bigger push to move away from c to languages that can prevent stuff like this from happening. this and just the pure unsafety. it's 2026. it's like pure negligence at this point, the equivalent of building a bridge with matchsticks because you like using it more.
How would using a language different than C help here? At it's core this bug is a logic bug, where "*" == crypt(x), because crypt in musl started returning "*" in some cases some years ago. This could be Rust. This could be Go.
Not every C vuln is a memory safety problem.
Another comment said that returning a * was to avoid returning null which would crash users of the library who didn't check for null. That seems like a C smell to me
18 replies →
>because crypt in musl started returning "*" in some cases some years ago
In Rust, the type of the return value would be a `Result<T, E>`, which means the chain of events that led to this would never have started in the first place.
6 replies →
> How would using a language different than C help here?
Easier to formulate invariants and check/prove them. Better language correlates with safer software.
I write firmware where the program has to run for years unattended.
You are right, C with manual memory management, no lifetime management, lack of phat pointers and a culture of leet has problems with deranged maddening bugs that other languages don't have.
But this bug first shows how inband signaling is bad news usually. But worse it's a failure to adhere to the spec properly. If you implement a library and you don't adhere to the spec you're going to bone someone.
Perhaps even worse is obviously the spec doesn't come with any standard testing framework. That was also the root of Apples infamous goto bug.
why does this logic bug exist? why are they returning a string, why not a result type, (nil, err), or something else?
oh right, because c is a crusty old language with barely any features.
Rust would not save you here as this is a logic bug. Nor would, well, really any other language. I suppose someone might claim that Rust's result type would save you but it really wouldn't: I can easily make the error type a String or pretty much any other type with little consequence.
Rust would absolutely have saved you here. Even with `Result<String, String>`, which is pretty poor design, there's no confusing `Ok(s)` with `Err(s)`. Sure you can go out of your way to write something like
but at that point I feel like the library is forgiven (except for the poor error design).
@my-huge-pony does make a good point elsewhere in the thread: if the potential error was not foreseen and the code was written originally to return `String` instead of `Result<String, Error>` then even Rust would be in a pickle.
It would, due to Result<T> and #[must_use] on it. Nobody would return “*” as a sentinel in Rust.
1 reply →
languages with more features would mean developers aren't choosing between unchecked NULLs or sentinel strings.
musl's crypt function could be written in Rust, Python, or JavaScript, and still return * when the key is too long.
What exactly do you mean "push"? What do you think is going to make musl developers switch languages?
it's unbelievable to me that there isn't a larger movement of developers to move away from c towards better & safer languages. look at how much pushback the rust4linux folks get, primarily from c developers who are simply afraid of change. imagine engineers not adopting safer building practices because "they prefer the old way"....it's disgusting honestly.
what will it take to move away from c? older devs retiring. Planck said that "science advances one funeral at a time" and i fear it will be the same for software.
3 replies →
guess what, this kind of sentinel value mistake could even be made in rust
no rust programmer would ever return a sentinel string like this.
6 replies →
Cars without seatbelts and airbags chopping off driver's heads in relatively slow collisions? Well guess what, that could happen in a modern car as well if the driver sticks their head out the window while going through a very tight tunnel!
1 reply →
[dead]
I often use "AuthenticationMethods publickey" which would disable all authentications except public key.
Isn't this safer than "PasswordAuthentication no", even if it's a different setting?
I would say that PasswordAuthentication=yes is by itself a vulnerability.
I run Alpine on hardware. I was not effected by this as while I do use ssh keys, there are no user accounts without passwords on my server.
> set PasswordAuthentication no in your sshd configuration and restart sshd.
Funnily enough, they don’t recommend checking pubkey auth is enabled and working. By blindly (or agenticly) following those instructions the risk is getting locked out of some machine.
This is brutal because a lot of docker containerized apps are very slow to update the base image if at all.
They shouldn't be including any sshd.
Docker images are generally not affected because they don’t ship ssh.
This looks terrible for musl, what a collection of terrible decisions on their part. I guess I'm staying away from it.
To err is human. There’s no escaping it, I’m afraid.
I agree entirely, but I'm not sure changing `crypt` to return `"*"` instead of `NULL` on failure qualifies. It's much more of a considered decision, which happens to be very wrong.
crypt glibc also sets errno to ERANGE if the string is too large, doesn't Musl libc do the same? OpenSSH should also add this check.
I presume Dropbear, thus musl-based OpenWrt routers are also not affected? Not that Dropbear has been an example in that regard.
There is this write-up/fancy page from the person who found the vulnerability[0] (HN discussion[1], how I found this thread).
At the bottom of that it says dropbear is unaffected.
> on debian, ubuntu, fedora or anything else with glibc? going through pam? dropbear? openbsd? you're fine
[0] https://news.ycombinator.com/item?id=50378228
No, the majority should not be affected.
Except in the case where their images are built with OpenSSH instead of Dropbear, which is the case for some manufacturers. (Turris, was noted as one such vendor)
Since after you posted, various people in the thread have reported dropbear is not affected
Here's an unrelated security issue I opened with Alpine earlier this year
https://gitlab.alpinelinux.org/alpine/aports/-/work_items/16...
It didn't give me the impression that I was dealing with a secure piece of software, it seemed like they were dealing with very complex issues while laughably failing at the industry basics that were settled years ago.
Similarly Ubuntu was also the only other distro that fucked this simple permissions config up, and lately with the latency to fix the TCP vuln, it seems to confirm that these are not reliable distros for the coming security era.
I keep saying that no one should be using alpine or musl, but people keep replying that static linking is totally worth it.
both alpine and musl are great in my opinion, and as far as I know theres no empirical evidence that it is more or less secure overall than alternatives. would you like to elaborate your opinion?
More evidence than allowing root to log in with any long password?
3 replies →
IMO only good thing about alpine is that it's a good environment to build stuff that links musl statically.
Never understood why Alpine is used to actually run anything in a container - if you statically link binary, then you don't need alpine as a base and if you don't what is the point? Don't @ me about image size, even without layer caching we're talking about tiny difference that is often dwarfed by size of your runtime.
Musl itself is a great example of taking "simple and lightweight" so seriously that everyone else gets to deal with the complexity. Slow allocator, underoptimized memcpy/string routines, limited locale support, no NSS, and endless glibc compatibility headaches.
But hey, at least libc is small. %%Proceeds to pull in megabytes of <any other>malloc, and other libraries to replace the slow parts of musl.%%
And now sshhole.com is a particularly spectacular example of what happens when "simple" meets the rest of the software ecosystem.
Static linking makes distribution easier, but it's an absolute security nightmare.
With dynamic linking, you just upgrade the library on your system and you're done. With static linking, every binary that uses the library has to be rebuilt.
So then rebuild everything? I don't see the problem. NixOS shows how it's done.
Docker is a static binary and HTTP is a new command line. I don't care what's inside, nothing good anyway.
I always use certificates with SSH. Saved me this time.
It's nice to see how quickly this is being addressed, and likely monitored for any unintended side effects.
Looking at the unpatched Ubuntu things in my life, and I'm pretty close to asking Claude to update things to see how quickly Alpine can get close, assuming responsiveness can stay this fast and quality.
[flagged]
[dead]
[dead]
Seems slightly confusing that the OpenSSH release notes page says:
> This release contains a critical security workaround that affects Linux systems that use the musl libc and PAM disabled.
But that there's nothing about it on their security page.
Even if it's not technically a bug in their code, the fact that they've made a "critical" release to address it suggests that it should probably be listed on the security page as well. But maybe (hopefully) they just haven't updated the security page yet.
musl libc violates the POSIX documented behavior of `crypt()`¹
> Upon successful completion, crypt() shall return a pointer to the hashed password; the first two bytes of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
Musl returns '!' or '*', not null.
¹https://pubs.opengroup.org/onlinepubs/9799919799/
Since OpenSSH is included in the base install, and enabled on OpenBSD - is it correct that they will need to update the famous:
“Only two remote holes in the default install, in a heck of a long time!”
https://www.openbsd.org/
OpenBSD is not affected, so apparently they are doing something right. But also glibc linux is not affected and pam linux is not affected, so it is only musl linux with the remote hole.
You became a victim of a very misleading post title. This isn't an OpenSSH problem at all, it's a musl problem.
I'm pretty sure the misleading wording is deliberate, OpenSSH is an excellent but non-commercial project which a bunch of entities would love to gain control of, in order to bug it with their spyware - thus the constant nagging on the bot-infested web.
The OP should be flagged.
OpenBSD doesn't provide musl-linked OpenSSH by default, so not yet[x2]
No, the bug is in musl but manifests in OpenSSH when compiled against it.
This vulnerability doesn’t exist on non-Linux systems, so not yet!
Workaround? Why not a fix?
AFAIU because the bug is in musl and not in OpenSSH
[flagged]