OK, I think I see what’s going on with this, in this configuration.
The goal from an attacker’s perspective is to get this to be true: `$ret = crypt($pw, $hash); $ret == $hash` when called by OpenSSH’s sshd. This is how sshd validates that the user logging in knew the account password.
The `$hash` is the “settings” to the crypt call and will normally look like “$6$--gibberish” for users with passwords set, but could look like “*” or “!” for users not intended to be able to login.
The bug is from the intersection of all of these things:
1. musl can return “*” for errors, rather than leave the return value NULL. Other libc have adopted something like this too, but seem to use a different result “*0”.
2. musl can also return “+” for errors, but only does so if $hash starts with “*”. This is intended to eliminate the possibility of an error causing `$ret == $hash` to be true, however it assumes that applications calling `crypt()` will set $pw and $hash to values corresponding to the `$ret == $hash` comparison to be made immediately afterwards.
3. OpenSSH can be convinced to pass the hash of a different user if the hash is “*” or “!” to avoid timing attacks that can enumerate the user list, so it will set $pw and $hash to correspond to different users. $hash will be a real “$6$--gibberish” style hash in this case, and OpenSSH expects `crypt()` to either return NULL or return a hash string. But now there’s two hashes OpenSSH uses. It does: `$ret = crypt($pw, $hash2); $ret == $hash1`;
4. This should normally not be a problem because OpenSSH will pass in valid $pw and $hash2 that should not lead to errors in hashing, so the $ret that musl generates should be a real hash that cannot match the “*” or “!” $hash1 for the login-disabled user. However as a DDoS mitigation, musl will return error if the password is too long when using SHA-2-based hashes (the default, it seems).
5. This error will result in a non-NULL result of “*” or “+”. Since OpenSSH passed in a real hash ($hash2), musl will return “*”, which compares equal to the actual user hash $hash1 for the disabled-login user.
The OpenSSL adjustment ensures a minimum hash length for the user before logging them in, even if crypt() succeeds somehow.
I think the conclusion that can be drawn from this is that the behavior of crypt() is howling-at-the-moon insane, and it should no longer be considered for any system built in this century, at least not without extensive wrappers that correct and normalize all these edge cases.
Not really. This issue is caused purely by the handling of the too-long input in the wrong way.
Firstly, why do crypt functions return a valid string to indicate an error? The problem is that the classic null return makes some programs crash which are not expecting it.
musl has a correct protocol for the error string, similar to the *0/*1 convention. It also uses two symbols, * and +. If the salt string is *, then it uses \+ specifically to thwart this issue: the program blindly assuming that the password is correct because the output of the function matches the hashed password field.
Where it seems they screwed up is this:
crypt("long .... string", "*") -> "*"
I.e. in this special anti-DDoS measure that checks for a too long input, they forgot to implement the key piece of logic that the output of the function, when an error is being reported, must be different from its second argument (the "salt", which is usually the entire hashed password or the contents of that field from a password file).
Thus the application ends up comparing "*" and "+", or "+" and "*" so the authentication fails.
Under no circumstances should "*" be returned when the second argument is "*", or "+" when it is "+", because it must be assumed that the app doesn't do any sanity checking, and blindly compares the second argument to the return value.
Is really the solution to a bug where a function had a wrapper that tried to solve its documented issue of returning *, to make yet another wrapper to try to solve the documented issue of returning true?
musl is actually safe against this use. OpenSSL takes a bizarre code path, detects that $hash is too short, calls crypt($pw, $other_string), forgets that it detected that the hash was too short, and checks $hash == return value.
I did note that musl would normally work just fine here. Though I appreciate that your other comment goes into much better detail about what was actually going on in the code, as my analysis basically stopped once I understood the high-level flow of what was going on.
Except OpenSSH qould have worked just fine if musl behaved correctly. Instead musl ignored the spec and tried to hack around badly written programs but ended up creating a security hole in a correctly written one!
Yep, the linked page doesn't /feel/ AI written (maybe it's using a grok or muse model whose tone I don't quite recognize yet?) but it kinda just states a lot of facts that make sense on their own but don't cohere into a whole vulnerability like the comment above does.
Because Claude wrote the linked page. Claude is impressively bad at communicating these days. with gpt-5.6+ approaching Claude level like 30% of time and Gemini much clearer but also the most annoying 7 year old ever, it’s a pretty lame environment especially for people who r new to everything
One has to remember the context here. The only error possible according to the SUS is ENOSYS, literally that crypt() is not actually implemented in the library. The traditional AT&T Unix crypt() was always implemented, and its encryption did not have failure modes, so there was never any error return to handle. This is why programs were written to never check the result of crypt(). It was only ever NULL on systems where one couldn't use the mechanism at all in the first place.
So the knock-on effect is that libraries that do have errors other than ENOSYS to accommodate, have to return magic 'impossible' values instead of setting errno and returning NULL, because of the weight of existing applications that expect nothing but the standard, traditional AT&T Unix (and indeed old BSD), behaviour.
Check the GNU libc crypt() doco. It defines a whole family of 'impossible' values that signal errors, by dint of listing a load of magic characters that should never be in the encrypted string. Including asterisk.
Also check FreeBSD's OpenPAM for a case in point. Even today, it does not check for a NULL return from crypt() and leaps straight to strcmp().
If you want to see real history here, check out the 1992 comment by Eric Andrew Young, that is still in the source for OpenBSD today (in des/des_fcrypt.c), on how SSLeay's crypt() had to be adjusted so that any single character password field, including "*", did not end up successfully matching itself.
It's worth pointing out that musl tried to do the same as SSLeay, and deliberately returns something different to the original salt so that they will not compare equal. It does not return "*" if the input is actually "*" in the first place, contrary to the headlined cartoon explanation. (The locus of the problem is the caller calling it with a salt that is not "*" and then comparing it with "*".)
Returning a string on error honestly is defensible (especially in a language like C where the type system doesn’t have exceptions or result types), the question is why the string was “*” instead of like “crypt ERROR: input password cannot be longer than 256 characters” lol
In-band signaling[1] strikes again. * should not have been used as special value, neither for "no password" in /etc/passwd nor "error while hashing the password" in crypt. I'm afraid we'll find many more of these abuses, specially in string manipulations. The null was ok, an exception (or equivalent) better, and Result<T> (or equivalent) best.
Exceptions are almost never the right choice. They create extremely complicated control flow extremely quickly, and then you have to test all the different control flows. Java software is an example of how this goes sideways fast. There's no way to create a class that prevents callers from reading the object before the constructor runs (in multi-threaded software, new first hands a reference to the caller, which can put it on the heap, then it calls the constructor), and objects are default initialized, so all fields (even static ones that are assigned in the constructor) can be null.
So, any field de-reference in Java can produce a null pointer exception. Try testing all paths in something like:
There are M * N * P paths. The only way to write testable code with exceptions is to panic on uncaught exception, and to never catch unexpected exceptions. Even then, you're relying on nothing unexpectedly throwing the exception type you caught.
I'd argue Result<T> >> NULL >> sentries >> exceptions for this reason. Sentries are terrible, but at least they're not computationally hard to use correctly.
I'd argue that's the fault of having default nullability, not the exceptions. You would have the same problem if every attribute access returned a Result<T> too.
And I'm not a fan of breaking control flow, but I still find it preferable to nulls. A null may travel far from the place where it was generated before blowing up, making troubleshooting harder, or it might come from a function you called for its side effect but forgot to check the return value. I gotta say I'm also a careful enthusiast of checked exceptions, it's great knowing which ways a function can fail.
But I think the consensus is growing that Result<T> (plus warnings for unused return values) is simply the superior choice.
> There's no way to create a class that prevents callers from reading the object before the constructor runs
That can only happen if you leak the reference (given we are talking about java language level semantics and not byte code), which I give you is unfortunately not that hard to do.
As for exceptions, I think they would pair beautifully with software transactional memory, which would work well with effects. In that case (and no side effects) it would not matter what code path we took, we could just go back to a known safe point.
Because ergonomy-wise I still think exceptions win (even though I do like Maybe types, and Rust's ? macro is plenty good):
- they let one focus on the actual happy path code
- no error is ever swallowed, unless explicitly written so (looking at you go with useless if errs that fail to properly handle the error), they are always handled somewhere
- they automatically "unwrap" the Ok result, and automatically bubble up the Err side.
- bubbling up is especially useful because most errors can't be handled at their source (e.g. IO error is meaningless, how you handle it depends on what you want to do in the first place, which is done several layers higher)
- they automatically attach a stackframe/source, while with result types you may get a list of Ok Ok Ok Err and have no idea how that Err got there.
> There's no way to create a class that prevents callers from reading the object before the constructor runs (in multi-threaded software, new first hands a reference to the caller, which can put it on the heap, then it calls the constructor)
Which is why the pattern of double-checked locking doesn't work.
You should never ignore exceptions! The least you can do is to crash the program. Then you fix the bugs via battle hardening until it no longer crashes. Making the program not crash by ignoring the errors will just mess with the state of the program until you have no idea what caused it. It's better to let it crash, load the dump into your debugger tool to see what went wrong, then fix the bug, and repeat until you have a stable program. Rather then just live by the motto "ignorance is a bliss".
Exceptions work fine and enable a clean separation of business logic and error handling. Visibility into partially-initialized objects is a language design issue independent of exceptions. In a non-exceptional language, the corresponding access to uninitialized fields would just be bugs, not errors you can see.
Nothing in your post actually supports the idea that exceptions are "computationally hard". The CFG error edges exist however the errors are spelled
PSA: you can typically declare different types of string so that your type system objects to mixing data of different types even though deep down we know they're both strings.
If you mean type aliases, that's a good idea within a single codebase, but the distinction between different types of strings doesn't survive serialization so it wouldn't have prevented this error. What they need is a way to signal errors out-of-band, like Result<T> or exceptions.
This is presuming that the type system is nomitive rather than structural and there is not serialisation/deserialisation or type refinement occurring. A structural difference is much more preferable.
This is a side-effect of Unix-y "it's all plain text" design that leans extremely heavily upon in-band signalling (among other design decisions).
If /etc/shadow was a JSON (or XML, even, although that would make me feel gross), this bug would have been much easier to see, because a field that reads '"hash": ""' sticks out a lot* more than a custom microformat that you don't understand.
With data structures, there's never "plain text" - it's all structured. The question is whether you want to use existing formats and tools, or create a new one for every tool and literally make footguns like this.
A few years back I un-bashed my life as much as possible in favor of nushell (though it's not the only shell to encourage structure). I feel like it has really helped me notice a lot of this-needs-more-structure type problems outside of my shell that I otherwise wouldn't have.
The elegance of the unix-y approach is actually that it's all bits. You can just join things together and shove the bits through, that's a wild simplification compared with having vendor-specific kinds of plugs. But we could have that kind of compatibility while also benefiting from structure.
So I encourage everybody to give a shell with structured output a try, let it rewire your brain, and then ask yourself: how is the OS holding this back from being everything it could be? Because if we build that OS, it won't encourage these problems.
When I imagine it, it would probe either side of a pipe to see if they support negotiation. If not, send the bits through in the same old unixy way. If so, use the newest schema for structured data that is supported by both sides.
The issue isn't the config, but the fact that crypt() doesn't have any way to signal failure. In better designed languages you would have exceptions, result types, or something. Even in C you have "return integer indicating success or error" which would've prevented the confusion.
Now the public API is hard to change but you could still avoid the issue by returning a longer token like "==DANGEROUS_CRYPT_ERROR_length==" or "==DANGEROUS_CRYPT_ERROR_unsupported_alg==" which would never match the password field unless someone was really trying to induce the issue.
Returning “*” instead of NULL to signal failure is the real footgun. Don’t blame the format.
For historical reasons, we could’ve ended up JSON as “hash”: “*” just the same. The /etc/shadow format lets you skip values completely, like “::”, so your argument doesn’t hold.
Terminating an app because of a NULL dereference is the other footgun. Sadly, we still have “panic” in modern languages like Go and Rust.
Random parts of it will be interpreted by the shell, others by cURL, or the filesystem, or the remote server. They each have their own encoding rules, nested. I count ten distinct separators and nine ways of escaping/tagging. It's one command. No, just no.
We have seriously overused strings and we're now paying the price.
At this point I’m resigning myself to think that we’ve just gotta wait for some people to die. It feels very dramatic and glib (no pun intended lol) and whatever to say, but at this point it feels like there’s a certain hostility in some parts to changing anything from the platonic ideals of however computing happened to work in 2004. “Oh we don’t need to mess with it!”
I'd say another UNIX-y thing - "C is good enough" - is part of the problem. C makes programming painful which results in all sorts of hacks.
E.g. in Pascal you can can define a record and then defined file of record to have typed I/O. I.e. it comes with built-in serialization format supported by compiler. Not a good one - but perhaps a step in a good direction.
OTOH C people starving for serialization often believe that `struct` byte layout made by a specific version of a compiler is a serialization format - a view not endorsed by the standard.
That would've happened to fix it, because "error_while_hashing" and "no_password" happen to not be valid password hashes. The right way would've been to have different syntax for hashes and non-hashes; you could've made it so that a hash starts with a '#' and if it doesn't start with a '#', the rest of the string should be ignored.
Or, for backwards compatibility, start non-hash strings with a character which can't appear in a hash, such as '@'. Your examples would then become '@no_password' and '@error_while_hashing'.
The vulnerable path in musl is only a little bit vulnerable, because "*" is not actually a sha256 encrypted configuration and should not reach this code: musl's crypt(..., "*") will not call that code! Turns out that sshd detects that the password hash is too short and does not forward the salt!
Look, no else statement! If that if doesn't get taken, then salt remains NULL. And the sshd code is really quite messy and doesn't keep track of the fact that this code path is inappropriate for validating an existing password hash.
A workaround in the sha256 path in musl wouldn't even have prevented the bug!
I'm going to give some credit to musl for unwisely using "*" and some credit to OpenSSH for having nonsensical control and data flow.
> (hey, this workaround is wrong if the salt is "x"!)
When the salt is "x", it'll return "*" and still not match, won't it? The code is correct as long as the second argument is the hash that's being checked. And like you say, the sha512 path isn't relevant since you only get there if the salt started with "$".
> Turns out that sshd detects that the password hash is too short and does not forward the salt!
I think you're misunderstanding the intent of that check. It was added by the 2016 change linked in the writeup[1] and causes OpenSSH's xcrypt() to select and use a salt from a different line of /etc/shadow to prevent timing attacks. It is correct for a crypt() implementation that returns NULL on error.
> When the salt is "x", it'll return "*" and still not match, won't it? The code is correct as long as the second argument is the hash that's being checked. And like you say, the sha512 path isn't relevant since you only get there if the salt started with "$".
I think you're right. That's what I get for doing a very very quick read.
> I think you're misunderstanding the intent of that check. It was added by the 2016 change linked in the writeup[1] and causes OpenSSH's xcrypt() to select and use a salt from a different line of /etc/shadow to prevent timing attacks. It is correct for a crypt() implementation that returns NULL on error.
I don't care what the intent of the check is. If you're doing a fake password hash to obscure timing, remember that it's fake and don't let the user log in! Don't deliberately pass nonsense into the live check and rely on the nonsense check failing to prevent the fake check from accidentally working. And don't have a code path that thinks it's checking the real password but passes in the wrong second parameter to crypt().
Or, better yet, have an entirely separate function fake_check_password() that does the fake check, returns void, and cannot possibly be confused with the actual password check code.
My general rule of thumb is: "no magic strings". This is much easier to implement in higher-level languages and when writing application/web code than system programming.
Don't return empty string/asterisk/anything else. Instead, return a failure value.
When absolutely needed, choose the magic string with more caution:
One string showing the SHA-256 of a non-existing hash? It's a good idea for it to be a valid hash. Now we have a new risk: if it's broken once, it's broken for all. So it's a good idea for it to be a random one for each machine and/or account. Even this might have problems I can't think of, but it seems to be as secure as an account with a real password.
If this is not a good decision for some reason, choose something unmistakable: "--ACCOUNT-HAS-NO-PASSWORD--". This is much less likely to be mistakenly used in a different context, is much easier to look up in a code base than "*", and is self-documenting.
One potential point of disagreement: it's easy to implement structural prohibition of magic strings in low-level languages too if you consider Rust a low-level language (which I've heard before, even though I didn't really consider it to be one for our application of it).
I'm not a Rust expert but do run a few thousand lines of it in prod for some perf/correctness critical code, and I wish it didn't have the reputation of being so difficult to comprehend. There's so much resistance to introducing it or new languages in our large org, that I'm surprised we were able to introduce our thin slice at all. I am not sure I'd use it where you need easy async primitives like goroutines, but it prevents whole swathes of bugs that are repeated over and over again with devastating impact, like the OP's bug, via thoughtful language design.
Maybe this resistance is an artifact of where I work more than large organizations generally, though. Curious about others' experiences. I'm not really an evangelist for the language but would really like to work in a more polyglot environment one day.
Above all, I’m a little sad that the musl developers decided to make a lot of “well it’s convenient” changes to core crypto functions — at least without the level of auditing that would have caught this! It’s tarnished their reputation in my eyes a little.
I think everyone is over fixated on AI finding complex bugs beyond a typical person's understanding either to locate or exploit.
I think for any one of those bugs there might simply be half a dozen footguns or bugs that once it's spelled out you wonder how anyone let it slide.
I think that's the case where LLMs might indentify most.
It's challenging though as the current AI audits produce so many false positives it's hard to know when it's being pedantic or when it's genuinely caught a human being a bone head
Finding more and more that these complex bugs people gloss over like you are now are being used in exploit chains and the end result is still my clients getting hacked
This one might be hard to find using the author’s approach of throwing everything at various builds of ssh.
It is not really that hard to find in the wild if you had a device built this way, when overflowing buffers on embedded devices is a standard attack vector.
It might be hard to root cause, but it would not be that hard to find.
Honestly the ideas around musl are interesting, especially being able to statically link, but given the sort of nonsense around how they handled pthreads, and generally other trade offs, it's hard to imagine wanting to use it over glibc (which everyone shits on) if you don't absolutely need static linking.
In many ways each individual decision makes some sense;
wanting to stop loads of programs being affected by a (remotely triggerable?) crash
wanting to avoid time based attacks
wanting to be able to set accounts to not have a password
Each one relies on a perfectly valid argument - that * can't match because we all know it'll be rejected by the other thing.
I would be more concerned about the lack of explicit return types, one of the arguments for strongly typed languages is that you should not have ambiguous data and yet…
Granted I mostly work with high level programming languages so I may be missing something.
I guarantee that every single developer of musl in 2012 had at least account on their machine with "*" as the password hash. Also crypt() is a standardized function and the standard states that a null pointer is returned on error.
That being said, OpenSSH did mismanage their attempts at handling a garbage return-value from crypt(), so there's definitely some blame to go around.
You must be new here. While I applauded the idea of musl when it first came out (coincidentally, glibc wasn't having the best time in its development then), the developer's arguments why <some behavior drastically different from every libc under the sun> was akchyually correct, which was why you had to fix a program that hadn't been broken for 30 years and now suddenly was, made me put him under the "clowns" category and never look at musl seriously again.
When later it turned out that musl's procrustian stack size made Python not really runnable under it, it cemented my resolve to not go near musl.
I’m also reminded of the clowns pushing “it’s smaller!” container images based on social media vibes rather than analysis where, yes, not using Debian+glibc saved tens of megabytes on the base OS but did so at the expense of measurable performance reductions & compatibility issues like the one you mentioned, and literally every single one of those dudes I worked with was shipping containers pushing a gigabyte so the size reduction had no measurable impact on deployments.
I am sorry, a little? I am deeply disappointed that libc decided to do the same just with another sequence.
And why because some dude was complaining sshd or whatever was crashing because it didn't handle nulls? Just handle the null my guy you already know why your bullshit is crashing.
Let's be real here, the main culprit is C. People love to say "but you can still get logic errors with Rust", and no shit of course you can. But it's much less likely. You wouldn't have gotten this bug.
A little?!? I’m sorry, but musl is a joke, this is yet another proof of that fact, and I have no idea why anyone uses it in production, they have only themselves to blame after the many warning signs.
In my last org I was in charge of internal testing and qualifying software releases for network equipment, and designing test harnesses and tests for these. Significant install base. All the major vendors. Many years. Basically doing the vendor QA for them, because we couldn't afford things to fail. As you do - I wish.
This was not security testing as such - mostly latency, performance and hardcore network "can we break it" and "does it comply" stuff, but also a lot of cable and module / line card pulls, pluggable and fiber breaks, random reboots and pre-empting a lot of possible operator inventiveness. At one point we considered SEU bit flip testing on rented cyclotron time.
And always we battled against the top brass urging us to speed up the process and don't bother with "really basic" testing, because we observed software quality going down over the years as the push for release and time to market stared to dominate.
So we always started with sanity testing - and that included oversized password and CLI input strings. Always. And we caught things more than once, but never like this.
This is why I don’t fully buy the argument that software that’s been around for a long time is more secure and reliable because it’s been battle-tested. There’s been a lot of time for cruft to accumulate until no-one can ever fix it because everyone is expecting it to function in the broken manner it’s always functioned. And some of the more obscure configuration options get a lot less scrutiny.
This is not an obscure configuration option. Your password file almost certainly has at least one account with "*" as the password hash.
Integration of different software is the stage most likely to introduce uncaught bugs. That is what happened here: musl intentionally broke the documented API of a function used by OpenSSH. Then Alpine integrated this incorrect implementation with OpenSSH.
I was able to reproduce this on Alpine 3.23.7 with OpenSSH_10.2p1.
Luckily PermitRootLogin [0] hasn't allowed logins with passwords for root since 2015 [1]. I'm guessing that's at least part of why this went undiscovered for so long.
Did anyone get it to work? I tried on the official alpine docker image, enabled password login for root. The shadow entry has ! as password by default, this didn't work with a > 256 character password (i also did passwd -l which also does !). I manually changed it to * and it still didn't work.
Ahh! Wasn't working on 3.16, which does indeed have an updated package.
But, got it to work on 3.15 - which requires changing the ! to a * in the shadow file in addition to other steps (making another user with a hash, setting PermitRootLogin to yes, etc.).
No, didn't work for me open OpenWRT x64 running in a container but that is probably 'cause I'm using dropbear instead of openssh-server. That's what OpenWRT uses by default so the impact there will be minimal assuming that most installations don't change to openssh-server - it isn't installed by default.
On top of OpenWRT on my router (running 25.12) running dropbear instead of openssh-server, I also do not have any accounts in /etc/passwd|/etc/shadow with "*" for a password hash and an actual shell (all are set to /bin/false).
In Alpine, these would all be the case for an out of the box install with openssh added unless you specifically install openssh-server-pam and configure sshd_config to use PAM (Alpine is already musl-c based and doesn't default to PAM).
In Unixese, shadow having ! means the account is "locked", right? I haven't tried the steps to replicate so maybe they did replace the * with a ! or something, but I would try making sure the account is not locked.
I can't reproduce it either. I used alpine:3.24.1 and openssh is openssh-server-10.3_p1-r2
I tried adding PermitRootLogin yes and PasswordAuthentication yes. Tried adding another user with a valid password.
Verified shadow entry has a *. Can't login as root or a test user. Tried another incorrect password first as well.
edit: can reproduce after `git revert 384e25e6bd1f2e7b30095900e48b570d46953510` in aports and rebuilding openssh without the fix. It also accepts the password on the 2nd+ attempt, NOT the first.
Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option because it means people actually notice that there's a problem. Silent compatibility hacks are the worst, they guarantee problems never get fixed and the stack of hacks never gets smaller.
Microsoft has historically been one of the leaders on the wrong side of this one, but they have also demonstrated how effective it is to call out badly behaved software and make it annoy the users with UAC. When Vista came out, so many applications needed elevated privileges to run because they were badly designed to write user data in system locations, but now their application was more annoying to use and came with a scary dialog about it making changes to your computer that other better-designed software didn't get.
More often than not if a program didn't actually intend to change system settings it was just a matter of badly chosen default paths which are pretty trivial to fix, but so many vendors just would not do it despite years of complaints from NT admins wanting to not have to spend hours with Process Explorer figuring out specific permissions to grant limited users enough access. Once their failures were put front and center, calling their application out as the one getting things wrong every time the user ran it, they suddenly had motivation to fix it.
For security it's better, but in many cases if you're not writing security-oriented software it can be worse from a user perspective. I remember the first job I had over 20 years ago was working on VoD streaming and one of the "tricks" to improve stability was mmap'ing the 0 page so that NULL dereferences wouldn't immediately segfault. Ugly as hell but when you were working with all sorts of more or less well coded media codecs and security wasn't a concern at all (worst case scenario the user would stream movies without paying) it made some amount of sense.
I suspect that Microsoft has the same history: the entire Windows platform was built with single-user PCs in mind, mostly only dealing with trusted input. The users were also expected to be tech-illiterate for the most part and unable to troubleshoot deep technical issues.
More recently I'm still working on video transmission, but on the production side this time. I also originally adopted the mentality that I should crash early if some unexpected situation was encountered, but I ended up walking that back a lot, because having a critical piece of live media production reboot mid FIFA worldcup final is generally Not Great. Degraded functionality is usually preferable. I still crash early in dev builds, but the production code just logs an error and continues if at all possible.
That said, I definitely agree with you for something like password handling. In this case, I do think it's warranted to crash the program without survivors if something is amiss.
> if you're not writing security-oriented software
It's hard to define what's security oriented and what is not. For example, is this HN web app security oriented? Not in the way of sshd but it must keep our accounts data separated and it must not let people exfiltrate data from the db, where for example it is storing the emails of some of us.
> because having a critical piece of live media production reboot mid
It's not great, but these days a trickster with AI will find some way to start broadcasting porn to a billion people because of the flaw, so may end up being the better option.
> Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option because it means people actually notice that there's a problem.
Agreed. I feel like this attitude has been holding Python back, too. It's one thing to criticize "move fast and break things", but it's also not good to ignore already existing breakage just because the pieces still fit together neatly, and it's not good to refuse to move at all.
I'd still rather have the software crash and alert a human that to continue in a faulty state. The way to add robustness in this case is to add backup systems, not "ON ERROR RESUME NEXT".
Is there a test or check I can use to determine that software is 'good'?
Oh there isn't? Good/bad is subjective? We need to scape goating hacks as bad software, as if anyone is capable of perfect code. Perfect code would require knowing the system inside and out, the stack up and down - no one knows all that, no one is perfect. Sufficiently advanced AI though can rip through all of that, find the weakest spot and zero day it, bringing the entire house of cards down.
Software you thought was good can still be hacked. There is no way to create provably unhackable software.
Which means all software is packed with vulnerabilities just waiting to be discovered, no matter how good you think it is.
Now everything installs to a user local folder, runs as the user, there are no UAC prompts, and malware can just freely tamper with other applications, act as the user, and harvest all data the user has access to.
Hey, but at least it can’t touch the System32 folder.
- (note root won't work by default because of PermitRootLogin prohibit-password default)
- attempt twice, first attempt fails second long password is accepted (that took me a while to understand)
In practice passwd -l seems to lock with a ! prefix, and new account without passwords are created with a single ! which is also not affected, I'm not sure how big the impact actually will be in practice and don't understand the recommendation to upgrade urgently right now... Just don't use `*`?
EDIT: ah, some default container images apparently have a default alpine user with that hash, so I guess that's bad for them. This makes a bit more sense now.
> some default container images apparently have a default alpine user with that hash
Do you have an example? Looking at Alpine's advisory, I can only see that their cloud images (i.e. what you use to run a VM, not a container) have a default alpine user, no container images. And even with those, cloud images with cloud-init are unaffected with default config due to `PasswordAuthentication no` being inserted by cloud-init into sshd_config. So that leaves... Cloud images that use Tiny Cloud for provisioning. That's an instance bootstrapper authored by Alpine from the looks of it so I'm not sure how many people would prefer that over cloud-init (which a lot of VM/bare metal management software may already make easy to use) or just a non-cloud image.
Well but, alas, a non-zero number of user probably do use it, though and "upgrade X" is a clearer and better instruction to communicate than "change this setting where maybe you don't even know what it means and it might break your workflow, but saves you an upgrade".
In some contexts (e.g. embedded) you'd be surprised at how reluctant some people are to upgrade when they can run through a dozen of hoops for workarounds such as changing hash instead...
It's amazing how similar this exploit payload is to the 20+ year-old Quake 3 vulnerabilities found by Luigi Auriemma (aluigi), which also used the exact same 256-to-1024 character range and nothing more than a long string of AAAA.... Here you're feeding >256 A's up to a 1024-byte limit to trip musl's defensive length cap, whereas back in the mid-2000s, Luigi was famous for publishing one-line .cfg scripts where the entire exploit was literally "say aaaaa..." repeated a few hundred times overflowing server's memory.¹ ²
A great example is Jedi Academy, a 2003 Quake 3 engine game whose multiplayer is still actively played today (its open-source engine port, openjk, is even in the Debian repos btw). Luigi had two classic scripts for it on his site: Jedi Academy <= 1.011 in-game server buffer-overflow (jamsgbof)¹, which crashed the server just by pushing a string of as past the 1024-byte console buffer, and Quake 3 engine in-game players kicking (q3msgboom)², where a wall of as hit the engine's 1022-char truncation limit and the leftover bytes confused the parser into disconnecting everyone on the server.
Kind of wild that 20 years later, the exact same payload goes from crashing Quake 3 matches from the tilde console to getting root on some SSH servers.
Offtopic: this is a very cool way to present commits and code excerpts. Love classic Windows styling.
Were those pictures with links generated using a publicly available tool that we could use, or were it custom (vibe)coded specifically to create this page?
To be honest, I love this sort of vibe-coding. Explains it really well, in an interesting way. If this is vibe-coding, then the AI has done a great job.
MITRE scored this as a 8.1 with a high attack complexity. [0]
I fail to see how the attack complexity of this vulnerability is high, as all it requires is attempting an ssh login with a password longer than 256 characters to an affected alpine image with port 22 open. Am I in the wrong?
Maybe that's on you though. I enjoyed it, though the discrepancy between the `*sshole` name and the `sshhole.com` domain made me twitch. How is anyone[0] obsessive enough to find this and write it up and yet not obsessive enough to make the amount of `h` consistent?
[0]: Be they human or LLM. Or an entity with any proportion of the two. I do not judge nor discriminate.
It has very low information density. This would read 10x if it were just a few paragraphs of text. Instead we need to render an animation every few sentences. Not a fan.
I had this classmate that had their ChatGPT instructions to respond in this same laidback, lowercase, gen-z style. i recognized the output as a lot of llm writing almost immediately
I do actually. My guess is that this was not just one-prompted, but someone took the time to refine it based on their tastes and expectations until it matched something they would actually like.
However, if someone already developed a strong AI-slop-tingle it will still be triggered by some telltale signs and let their subconsciousness know that they can't possible enjoy this.
I think at some point I'm going to get banned from HN for repeatedly saying this, but here we go:
This thread highlights why I often say that at some point in the future, the obsession with whether something was written by AI or not will be classified as a mental disorder.
It is tiring to virtually every day see this discussion play out in some HN thread or other. It's also just noise. I don't think your comment, nor the thread it inspired, added anything of value to the discussion.
Yes, correct for OpenWrt. OpenWrt defaults to dropbear for both server and client. However, more advanced users may optionally choose to replace it with OpenSSH, or install OpenSSH side-by-side, to gain additional features of OpenSSH.
To be fair, I have less than zero sympathy for anyone who runs SSH on any platform without `PasswordAuthentication=no` in 2026.
Its not asking for trouble, it is there will be trouble, not if but when.
As for the people who say "well, I run SSH on port 942329053052". Well guess what, the bots make short work on that. We tried it at $work as an experiment, multiple bots found it in less than a couple of hours. That's why your SSH config must be hardened, preferably running on a hardened OS too.
> Its not asking for trouble, it is there will be trouble, not if but when.
Did any of those bots brute force a password? Because a 20 character random password is comparable in brute force complexity to an SSH key. The experiment you ran was just as valid for SSH keys on that port, and had no bearing on password vs key.
The point is they found the security by obscurity so-called "hidden" port in less than a couple of hours.
At that point, what they did is irrelevant. The point is that security by obscurity does not work in 2026.
> Because a 20 character random password is comparable in brute force complexity to an SSH key.
You can't be serious with your comment.
No its not.
And even then, SSH key is asymmetric, you're not sending the private key to the server. But with a password you are, which leaves you vulnerable to many attack vectors that simply do not exist with keys.
How is password auth weaker than key auth? Bugs in auth code can happen in both pathways. And apart from stupidly short passwords both should be equally impossible to brute-force.
In other words, conceptually, at the end if the day, isn’t a key just another name for a password, really?
Password gets sent to the server. Private key does not get sent to the server, and authentication protocol can be forwarded safely, enabling things like ssh-agent, FIDO SSH keys. This is solvable with passwords (see: augmented PAKE), but `PasswordAuthentication` isn’t that. But there are also settings where none of this really matters, yes, to be fair if that’s what you’re referring to.
What people think of as a very strong password often has <100 "security bits" of entropy.
Keys are usually at least 128 bits of entropy, and it's easy to take them up to 256.
Yes, both are a shared secret, but password auth - even with challenge-response - is weaker than key auth exactly because key auth provides more guess resistance.
Even if you don't want to go all the way to SSH certificates, SSH public-key authentication is laughably easy to do.
I also don't buy the argument "but its on a ULA address". Attacker's old favourite these days is finding a way in (North–South) and then using that as a stepping stone (East–West). Also the general concept of "lower security is OK because its on the LAN" is so 1997.
Bug > are password logins still on? they are by default
You > This particular bug works even with password authentication turned off.
No. The bug is related to password authentication. If you do not have password authentication then you are not affected.
What freaks some people out is they chose parameters where they do have password authentication but they expect to always fail. That's a bad choice, like OK you did fit a cheap manual keyhole, but you threw the keys away thinking now nobody can open that - you realise somebody with a cheap jiggler can open it right ?
I mean there are 65k possible ports so it's the equivalent of guessing an 16 bit password, i.e. two (extended) ascii characters. It's not very surprising that 'multiple bots' found it: a single `nmap` command will give you the open ports in a few seconds.
What is out there that uses musl without PAM and hosts an OpenSSH SSHd with PasswordAuthentication?
Possibly public SFTP servers. I have allowed null passwords in the past for anonymous public SFTP and I run Alpine Linux exclusively for my servers both VM's and physical. The shadow package (PAM) did not get installed by default. I installed it but I had to specify it in the installation so there are likely people without it. There is probably a way to find them on shodan.io.
I will activate one for people to play around with for today:
sftp hn@ping.ohftp.org
There's no firewall and no password. Hack away HN.
Edit: Sure, a small minority presumably run sshd under these conditions. But a small minority times the number of Alpine containers running in the world gets to a large absolute number.
Edit 2: You really don't need to look very hard to find public container examples at million-download scale.
Many, many Docker containers rely on "docker exec" or the kubectl equivalent to gain CLI access when needed. To have sshd in an Alpine Docker container, you're generally need to be running init and a bunch of demons, not just a single standalone service.
I'm guessing in practice relatively few people are actually exposed. But if they are, it's catastrophic.
The terrifying decision for the API to return "*" on failure is a second-order consequence of memory unsafety, because it was a half-baked attempt to mitigate a common source of UB that people were encountering in the wild.
Wow. I wonder which of the common fuzzers would have found this a decade ago. If literally any password over 256 chars works, that means nobody has ever fuzzed openssh on musl linux
In the TXR Lisp run-time, I spent a disproportionate amount of time on the crypt wrapper, making sure that it detects errors on numerous version of crypt, and converts them to an exception.
I settled on this. There is a block comment on the error handling, dating back to just 2023. There is a tiny infelicity in the wrapper in that we rely on the value of errno in the error path, but we do not set errno to zero and check for it being nonzero (evidence that the function set the value).
Applications using crypt directly can do the same thing: if it returns null, that's an error, of course, but don't use the result if it is less than 13 characters: that's also some kind of error token. The classic crypt() puts out a 13 character hash. Everything else is longer.
static val crypt_wrap(val wkey, val wsalt)
{
val self = lit("crypt");
const wchar_t *cwkey = c_str(wkey, self);
const wchar_t *cwsalt = c_str(wsalt, self);
char *key = utf8_dup_to(cwkey);
char *salt = utf8_dup_to(cwsalt);
#if HAVE_CRYPT_R
struct crypt_data *cd = coerce(struct crypt_data *, chk_malloc(sizeof *cd));
char *hash = (cd->initialized = 0, crypt_r(key, salt, cd));
#else
char *hash = crypt(key, salt);
#endif
free(key);
free(salt);
/* libraries cannot agree on how to report unrecognized or bad hashes:
*
* - older glibc versions, other libraries return null
* - libxcrypt, integrated into newer glibc puts out two
* possible failure tokens "*0" or "*1", documenting
* that an error token starts with "*" and is less than 13
* characters long.
* - musl uses "*" and "x", the latter being in the valid hash charset!
*
* let's go with: null or less than 13 chars means error.
*/
if (hash != 0 && memchr(hash, 0, 13) == 0) {
val ret = string_utf8(hash);
#if HAVE_CRYPT_R
free(cd);
#endif
return ret;
}
#if HAVE_CRYPT_R
free(cd);
#endif
uw_ethrowf(error_s, lit("crypt failed: ~d/~s"), num(errno),
errno_to_str(errno), nao);
}
Never attribute to malice that which yada yada yada
And like who even needed to plant back doors? If you're the NSA you already had an agent swarm in 1993, but it was made out of dudes from Berkeley and New Haven instead of racks in Prince William County and 印西 (and they were literal secret agents I guess, pretty metal). You can get back doors just by foraging, no need for agriculture.
Pretty big? For this purpose, my instinct would be to select an ASCII character that's not alphanumeric, not directional, and not a common separator, and that's like, what, a set of 11 characters?
You have considerably underestimated both how fallible humans are, and how high their tolerance is for risks which aren't immediately visible.
This isn't even just the software industry, humans are like this about everything. Imagine explaining to something rational that (a) humans didn't realise that they won't be able to competently operate a motor vehicle while watching TokTok dances, and (b) when we finally made that illegal because it's such an obviously bad idea they would keep doing it anyway.
This is why those "What if it builds SkyNet?" conspiracies were so stupid. Why on Earth would you build SkyNet when you can just invent Weeaboo and the idiot humans will die playing ?
True, but also, that is exactly the pretext that would allow someone to maliciously, intentionally commit a backdoor.
There was even a contest for hiding/obfuscating malicious code, e.g. as an honest mistake: https://www.underhanded-c.org/ . And what do you know, the most recent, common approach shown on the page, is: improper comparison inputs that give unexpected comparison results, which is... well... this submission.
I expected let another AI slope website and instead that was a real human writing about what they found. Sure they used LLMs to find this, but they used their real fingers to write this. See, this isn’t that difficult, and it’s a lot more enjoyable to read.
I think the following should be done. Either this, or eventually DataKrash will happen. International coordinated emergency should be declared with an immediate order to shut down all non-critical internet exposed services. Critical services should be put under armed guard and remote access should be disabled except for whitelisted critical staff. Then lustration of programmers should begin. Everyone who ever coded in C(rime) language and any descendant languages would be barred from this profession at least for next 30 years. Then maybe new software will be written with due care, and any new services should be put online only when written without any C - no OS code in C, no library code in C, no C conventions, etc
I avoided using alpine-based containers ever since finding out a lot of Linux server software is actually Linux + SystemD software. The union set API of both Linux and SystemD seems to be the default "Linux API" today.
Container images rarely ship with an init service of any kind (systemd or otherwise); the expectation is that your application is the root process. This includes Alpine-based images.
That's their point. The software they want to use often assumes its being run on a platform with systemd, so running that software on Alpine which makes a point to not use systemd causes challenges for them compared to using a different distro.
Edit: My bad as pointed below this is not in OpenSSH but in musl and it appears in the relase notes https://www.openssh.org/releasenotes.html#10.6p2. I had scrolled straight to the bottom of the website because the animations annoyed me. Still would like a CVE for proper tracking though!
OpenSSH considers this to be a vulnerability in musl and not OpenSSH, but shipped a fix anyways since that helps mitigate impact... Whether you agree with that or not, it's up to you :)
The last few weeks have made me feel very good about my policy of zero internet-accessible ssh ports, ever. On AWS, use SSM. For a VPS or home server, access everything over WireGuard.
Such solutions always felt like swapping one risk for another. With good separation (running the VPN on a separate host from the SSH login, otherwise a bug in the VPN solution still might result in a shell on the box), at best you'd need two vulnerabilities before someone gets in, but it's not fundamentally different unless you find a way simpler protocol where you can be more certain that it's secure than ssh is (for all its scrutiny and track record of no pre-auth bugs for decades)
Putting services behind ssh forwarding was the way to secure something you're worried about rather than the other way around, but Wireguard actually makes a good argument nowadays. Iirc they did proofs for the cryptography and it has good design in terms of attack surface, and it's old/mature enough now that I'll say it's probably better than exposing SSH. On the other hand, when this thing running in your kernel gets compromised, you're slightly worse off than with someone logging into a passwordless service account as in the current bug
One thing that makes me feel a lot more confident about exposing a WireGuard port is it’s completely invisible unless you have the host’s public key. Nothing will show up on a port scan.
Its security model doesn’t rely on that fact of course, but it’s the cherry on top.
Thirty-odd years ago, you could get to a Unix shell from user terminal session of the Vancouver Public Library (British Columbia, Canada) by entering a super long line.
From there, even though you were not root, perms on important stuff were wide open: rxwrwxrwx.
the more i read about this, the more i'm convinced, contrary to what seems to be the prevailing opinion here, that this is primarily an OpenSSH logic bug.
which is not to say that crypt() is well designed or specified - it's clearly deficient. but I can't fathom how the OpenSSH maintainers could come to the conclusion that the root cause is "outside OpenSSH itself" when the root cause is that OpenSSH's mitigation of timing attacks rested on faulty assumptions about the return value of crypt().
A common complaint I've seen (and I experience on occasion) is AI agents being a bit silly in their overprotective coding style. They nit pick and catch every single little error. This can be frustrating since a lot of the time they are trying to handle edge cases that are so unlikely to happen, yet to cover them requires significant additional code.
This might be the beginning of the reason we shouldn't be reading the code. It's like in chess engines where the players complain about how weird, alien and unintuitive the moves are. But 100 steps down the line the engine finds some edge case exploit that gives it enough advantage to win.
So we may be crossing a line, right now, where the non-intuitive alien code being generated by agents is seeing edge cases down the line that might allow exploits. And if we "correct" the agent, we may be leaving ourselves open to exploits we don't see and have difficulty understanding.
The answer is not to reduce our understanding. That is the worst possible conclusion we could draw from what we're seeing. The answer is to increase it.
I am saying that we won't be able to write it, in the same way that Magnus Carlson (or whoever your top chess player would be) cannot compete with a chess engine.
I am arguing that, much like the "best" chess move is unintelligible to best chess players, truly secure code will be unintelligible to humans.
Tangent: Pangram says the text is 60-ish% AI-generated. Not that surprising given the look of the page and the work being done with AI in the first place, but it is interesting to me because this doesn't look like your standard wall-of-text AI purple prose. So it's notable that a) you can generate convincing text in this style and b) Pangram detects it anyway.
PSA for OpenWrt users: If you are getting the post quantum message on login which points to openssh.com, you are likely using dropbear nonetheless. Check in System->Software. I was confused and worried why I seemingly had the vulnerable OpenSSH installed, but its just dropbear using an OpenSSH link.
The bugs and commits that "fix" them reveal somewhat of a shitshow for how musl is managed. All projects have bugs, but I can't help but question why the wheel was reinvented here. Many toes have been stubbed along the way.
Nice web site. However, I've got a server with 14 alpine-based docker containers in production. Among them 13 are widely distributed images (like web server, databases, etc.) and one is a custom build image. Only one of them have an SSH server publicly exposed, Forgejo, but it's its own internal SSH server which only accept public keys and Git commands. On the internal network, only 3 of respond to SSH. I haven't tweaked their default SSH config. None were vulnerable to this attack.
I tested an unpatched forgejo (enabling passwords temporarily, and even adding root to permitted users) and couldn't trigger this.
ETA: I could trigger it. You need to enable passwords and add a second user with a valid password, then git@ works. Obviously this still means the default forgejo is safe as you say.
Did we declare "break the world Thursday" and nobody told me? I feel like it's three weeks in a row that we've had a once-in-a-decade security issue pop up.
I have alpine linux on an arm device, and I have all the prerequisites for the exploit, yet I am unable to exploit myself. All of the Who is affected? from alpines security advisory applies to me. What am I missing?
> except a lot of programs never check for NULL. they just compare whatever crypt() gives back with the stored field, hit the NULL, and crash
> so musl (the libc under alpine) goes ok fine, crypt() just won't return NULL anymore. if it fails, it returns a single character instead.
Considering it's been accepted that crashing if malloc returns a NULL is your fault, did musl really feel the need to accommodate bad implementations like this or is this just AI speculation?
"unfortunately, a large portion of programs which call crypt are not
prepared for its failure and do not check that the return value is
non-null before using it. thus, always "succeeding" but giving an
unmatchable hash is reportedly a better behavior than failing on
error." [1]
Such a questionable decision! "Rather than let a bug in a caller (not handling the valid null output and thus crashing) surface, let's just silence it and pretend that the caller is correct! A potential clash in some situations (resulting in this vulnerability) is better than a crash!"
Maybe I should update my plea to fail loudly (https://alejo.ch/3gk) to list this as an example.
> did musl really feel the need to accommodate bad implementations like this
I don't know about this specific case but in general? Yes, totally. No one is going to fix all existing software that depends on libs behaving exactly as glibc, you have to accept some trade-offs. Either your new implementation stops being perfectly simple and clean, or no one uses it.
Thanks you to the team that found this. Exceptionally well explained and funny at the same time. What is most important is how people just checking stuff( remember the .xz hack that was tried?) has save the internets ass so many times. We should give good security researchers UBI for life for just doing good work.
It would be nice if the passwd crypt functions didn't have timing dependence on password length. Then there'd never need to be hacky workarounds like what musl did, and various tricks to avoid exposing whether users exist on systems.
Man, that author is really committed to that lowercase aesthetic. Even the entire personal website. I wonder what's going on with that? We used to do that back on AIM informally. I mostly just see Gen Z and young kids still doing that now. Pretty crazy how all the stuff that used to be trendy comes back.
Rust tells you off if you don’t handle Results. Sure, you can still just unwrap() it and have it panic, but at least that’s a specific decision instead of forgetting to check for NULL like in this case.
Not the worst site, but it's clearly a publicity machine from wakefield.co, whom I haven't heard of.
Also, exploitation conditions are super narrow, albeit, any password is wild.
Can we just have a site with the text and images without the awkward scrolling crap? Hijacking basic features is an instant click away from sites for me
Edit: It seems to work fine on mobile. I'm not sure why the scrolling behaviour was added on the desktop version of the site
Can we all marvel at the site. The author says they did the discovery on November 3. Since then they apparently bought a domain, set up the webpage, created animations, and a cartoon. All kinds of stuff, and got it to explode on hackernews. A span of 2-3 days. How long ago was this not even conceivable?
It's shocking that it took so long with the amount of investment and hype you'd expect this kind of bugs to be already fixed, but it still need a human to tell the agents what to do, which could also work with any fuzzing framework
Am I missing something or is this written up to seem a lot worse than it actually is?
It starts with a bold claim: "ssh accounts that should not accept a password will accept any password between 256 and 1024 characters". But then it keeps adding qualifiers:
- sshd compiled with musl
- not using PAM
- password logins enabled
- at least one account with '*' as the password "hash"
- at least one OTHER account with a valid password
- server is aligned with 2 ley lines
- hacker is wearing a blue t-shirt and green socks
Ok, the last two are an exaggeration, but still. The conditions are all ANDed, miss a single one and you're fine.
On top of that, as someone else in these comments reminded, the target account needs to have a valid login shell or other exploitable program assigned to it, and these passwordless accounts tend to have nologin or something similar.
To me this seems to fall on the "technically possible, practically very unlikely" bucket, kinda like the PHP vulnerability from the other day. Maybe this has gone unnoticed for so long because it just doesn't happen.
How many alpine installs are running sshd? Most containers using it as a base image aren't. Embedded Linux is unlikely to be running it, too. And if one IS running sshd, how often is it with unchanged defaults? I don't buy the inference here.
I'll concede that embedded linux might be my blindspot here. Maybe cheap web hosting, too, but that seems much less likely to be using musl.
For what it's worth, default Alpine installs don't have an account with * as its password hash if you use the root account as your main one. Not sure if that's also true for ones where you set up a user, maybe those ones are vulnerable.
I'm genuinely curious what's the Venn diagram here because I feel a lot of embedded systems precisely have one condition but not the other, e.g. password yes but not necessarily multiple accounts, e.g root only, while using musl. Namely almost all for almost all, but if only one is not overlapping another then it's actually very few.
does this really matter? It could be used to do something devastating. The number of systems that match those requirements is only one factor. If just a handful of those are highly critical for humanity it could lead to disaster. Eg. hacks into fridges, centrifuges etc.
- blue t-shirts and green socks have been added to the default loadout for new hackers, since it's optimal according to AI and costs nothing to the attacker in terms of compute
- there are so many ley line "maps" out there that basically any server placement counts as aligned with at least two, so this one's free also
Disabling sshd password logins is security 101, and the default for some cloud provided images these days. If password logins haven't been disabled, what other doors have been left open anyway?
This always seems bonkers to me. I think SSHD should give you ONE password free connection of setup to copy your key over and then it just shuts the door behind it.
Alpine Linux (which uses musl) images are popularly deployed by those who use docker, and apparently the issue is there on the default install.
Not sure whether the PHP vulnerability actually had been exploited, but I'm pretty sure there are literally thousands of machines out there affected by this ssh issue right now as we speak.
This isn't going to end well.
---
Addendum (which may or may not be related to the above) --
Everyone has heard of the tale of the boy who cried wolf. The main moral of the story is not to troll your fellow townsfolks so they don't distrust you. But everyone with a sane mind gets that.
The (IMHO) under-appreciated lesson is actually: don't be those villagers. Believe stories you are given, even if you might look like a fool for believing in them. Don't assume what happens is the same as what happened in the past. Look at the facts yourself. Trust and verify.
Alpine in containers is usually not affected. Most images/containers haven't ssh/sshd installed at all. Remote access is usually done from the host system.
> kinda like the PHP vulnerability from the other day
What? Pretty sure Cloudflare detected widespread attacks almost immediately after it was published. We are probably lucky that not much more happened because of responsible disclosure and the large media attention led to widespread admin awareness.
It’s irrelevant how widespread it is. It’s a bad bug that should be fixed. Why do people always say stuff like this? If you’re running a system that is exploitable then you deserve to know about it and to be provided a fix. It doesn’t matter all that much how widespread it is. The CVE system doesn’t exist to cater only to highly exposed systems.
It matters how widespread it is because humans have limited time and resources to solve problems. If these conditions only exist in 0.001% of installs, is it worth the man hours to fix it? Alpine may think not, because like others have mentioned, there are likely other higher probability vulnerabilities being exposed in order to meet the conditions for this to even matter. A judgement call can (and clearly has?) be made to prioritize other work.
The good news is that considering Alpine is open-source, you can be the one to make the judgement call that it is worth fixing and go out there and fix it ;)
Openwrt is mentioned to be affected and we use openwrt in our product. Looks like our configuration is different enough that we are unaffected, but still I thought I mention the vuln in our company chat. Turns out that M$ Teams does not let me post the link. Like in China, the party knows what news are good for you.
I'm starting to realize that trusting some random people who wrote the gruddiest, boringest, most torturous low-level stuff might have been a bad plan.
At this point I feel embarrassed in front of the AIs.
This is a super duper annoying site for people who just want to know if their systems are vulnerable, after patching like a dozen critical vulnerabilities in other software over the last week and are almost at the breaking point of resigning.
Can we have a “don’t treat me like five/toddler” / “idgaf about the explanation” quick link to a section for adults who need to patch systems?
(And if your reaction is “just scroll down, stupid”, I’m happy to report that with the scroll jacking, I scroll not enough / past the critical sections nine out of ten times.)
Thanks for mentioning that before I tried to scroll down. Forewarned, I was able to click on the "2026" link once the left-handed "scrollbar" (not the right word but I can't figure out what the right word should be) appeared, which let me jump past most of the verbiage and get to the "are you affected?" section just a couple pages later.
What this tells me is that musl+openssh-server is an incredibly uncommon setup, as "at least 257 character string" is literally the third thing I try after "admin" and "12345" when greeted with a password input.
This AI slop write-up is so fucking annoying to read. I don't even mind the AI generated website. Why can't you even bother to spend like an hour to properly sum up the exploit yourself?
I think it varies from person to person. Since I'm kind of from the TikTok generation, I don't have much trouble reading this kind of animation, but some people prefer static sites. That's just how it is.
I also found scroll-triggered animations hard to read at first, but these days a lot of sites use them, so I've gotten used to them.
Yep, scrolling through my phone and reading slowly to understand. I'm seeing distracting animations below what I'm reading. And when I go to them, there is no way to restart them. So it completely misses the point.
Am I the only one who is shocked that such important and security-relevant software just does unorthodox stuff like "well let's just return `*` if it fails" and other kind of "hacks" ?! The developers must know that this is very bad software engineering and will definitely cause issues at _some_ point in time, no?!
Shouldn't the title of this post be *sshole? An asterisk isn't disallowed in post titles, is it?
He mentions that circa 2019, Debian's libcrypt stopped returning NULL and began returning the 2-character string '*0' - he doesn't elaborate, but this was part of a larger migration by many distros. glibc deprecated its own libcrypt so a more agile project that 'could move faster' would take its place [0]. This migration happened in Fedora 28 [1], Arch [2], and Debian bullseye [3].
Unless you specifically compile it with --disable-failure-tokens, on an error libxcrypt will return that 2-character failure token string. [4] [5] This behavior is very old [6], and hasn't been POSIX-compliant since at least POSIX.1-2001, which is based on SUSv2, which is from 1997.
So they've knowingly kept this non-standard backwards compatibility cludge for almost 30 years, and continue to claim it "isn't well standardized" in their man page despite POSIX and glibc existing, but at least it didn't bite them in the asterisk like it did musl. (Yet.) There's a new issue about removing this behavior now. [7]
something interesting to note here is that while this isn't a memory safety vulnerability, a modern systems programming language with better error handling could have easily avoided this whole situation. error handling in C is a PITA, and we're paying the cost of decades of pretending that it's not a problem
All that site is lacking is marquee and furry creatures. AI slop sites are so horrible with their blinking animations, l33t dark background and chaotic text.
It’s dumb and negligent to be missing “PasswordAuthentication no” in 2026. It’s been so for at least ten years. OpenSSH should have turned off password auth in the defaults years ago.
> how did I ran into it? [...] i had agent swarms. a lot of them. let's not get into it
Can we please now normalize AI usage in development? Just some other day people asked for "NO AI used" badges for software as measure for quality, and praised i.e. NetBSD for being AI-free. Yes, we come from a phase of AI slop, but it has become obvious that there are plenty of legit usecases where AI beats human-only effort nowadays.
Not sure why but a lot of people seem to be missing that you don't have to look at AI as a binary "you're either with us or you're with the terrorists."
Just do HITL (human in the loop). Use an LLM to help you author, but steer the AI in whatever direction you want. It's like having a slightly delusional copy of yourself to pair program with. If you take the time to provide an LLM with some additional skills files and the like, with you in the pilot seat, you can get solid results (it doesn't look as cool as a hands free agent setup but the quality far surpasses anything they can do solo).
> Can we please now normalize AI usage in development?
1. It is already normalized outside of a tiny, but very vocal bubble that lives mostly on Bluesky (sometimes also on Reddit/HN/Mastodon).
2. I don't think that even the most staunch AI hater would object to using AI for security research as long as all the artifacts (code, docs) that were produced based on the research are human-written. "Yes, there is a bug in my program, but I refuse to fix it, because I know that you found it using AI" would be a pretty weird position. Not sure, how would you defend it.
I feel like there is a lot of pushback from delusional engineers, that won't accept their coding skills have been rendered useless - so they come up with tons of reasons to avoid AI, because it just has to be bad. Anything to hold up the ilusion you are an well-paid white-collar worker that can pick employers anytime. And I say this myself coding since the late 90s and making a living off it - it is a painful pill to swallow, but the genie does not go back in the box so just accept it and move on.
what was done to report & patch the vuln? Why publish it in this format? It’s presented like a whistleblower who’s exhausted every approach, but the effort to do the right thing is not clear.
> on alpine's official aws and azure images that's the default alpine user, and it has passwordless doas. so it's root
good fucking god. if I get this correctly... you can insta-own any aws/azure instance that was set up with alpine as long as it has ssh enabled, which it probably has.
Can we hear something from those who say that using port-knocking is "security through obscurity"?
Really: from the XZ backdoor in SSH (which went live on some distros) to this...
What's you rationale for calling port-knocking "security through obscurity" and not "defense in depth"?
Put it this way: if I advertize that my server is using port-knocking, that the sequence changes every 30 seconds (like TOTP) and that as soon one sequence is used, it's unusable (so you've only got one attempt to SSH in every 30 seconds, even if you know the sequence)...
Where's the security through obscurity?
But really: even if had musl and PasswordAuthentication allowed (I don't have neither), how would an attacker gets through the port knocking?
More fundamentally the question is: in which way is your system that does not use port knocking more secure than the one who do?
(remember: we're talking about remote exploits by simply having exposed TCP ports open or SSH listening)
Wanna have a talk too about the "solutions" proposed by "security experts" here "because SSH is too old" and that happen to be crufts on top of SSH that had remote root exploitable backdoors? (sorry, holes)
Disclosure: I work at Detectify. A few practical notes for anyone checking their own machines:
- It affects Alpine's openssh-server (built without PAM), or openssh-server-pam with UsePAM no, when PasswordAuthentication is enabled. Alpine says every release they checked is affected, 3.16 through 3.24 and edge.
- Fixed openssh packages are out for all of those, including the older 3.16-3.20 releases. Upgrade and restart sshd (rc-service sshd restart). If you can't upgrade yet, set PasswordAuthentication no.
- To check a box: apk info -v openssh-server for the version, and sshd -T | grep -iE 'usepam|passwordauthentication' for the effective config.
- Containers built FROM alpine that run sshd are easy to miss in an inventory. So are the official tiny-cloud images (3.18.11 through 3.24.3), which ship an alpine user with no password and passwordless doas, so a bypass there means root.
- After patching, look in /var/log/messages for sshd lines saying "Accepted password for" an account that has no password.
OK, I think I see what’s going on with this, in this configuration.
The goal from an attacker’s perspective is to get this to be true: `$ret = crypt($pw, $hash); $ret == $hash` when called by OpenSSH’s sshd. This is how sshd validates that the user logging in knew the account password.
The `$hash` is the “settings” to the crypt call and will normally look like “$6$--gibberish” for users with passwords set, but could look like “*” or “!” for users not intended to be able to login.
The bug is from the intersection of all of these things:
1. musl can return “*” for errors, rather than leave the return value NULL. Other libc have adopted something like this too, but seem to use a different result “*0”.
2. musl can also return “+” for errors, but only does so if $hash starts with “*”. This is intended to eliminate the possibility of an error causing `$ret == $hash` to be true, however it assumes that applications calling `crypt()` will set $pw and $hash to values corresponding to the `$ret == $hash` comparison to be made immediately afterwards.
3. OpenSSH can be convinced to pass the hash of a different user if the hash is “*” or “!” to avoid timing attacks that can enumerate the user list, so it will set $pw and $hash to correspond to different users. $hash will be a real “$6$--gibberish” style hash in this case, and OpenSSH expects `crypt()` to either return NULL or return a hash string. But now there’s two hashes OpenSSH uses. It does: `$ret = crypt($pw, $hash2); $ret == $hash1`;
4. This should normally not be a problem because OpenSSH will pass in valid $pw and $hash2 that should not lead to errors in hashing, so the $ret that musl generates should be a real hash that cannot match the “*” or “!” $hash1 for the login-disabled user. However as a DDoS mitigation, musl will return error if the password is too long when using SHA-2-based hashes (the default, it seems).
5. This error will result in a non-NULL result of “*” or “+”. Since OpenSSH passed in a real hash ($hash2), musl will return “*”, which compares equal to the actual user hash $hash1 for the disabled-login user.
The OpenSSL adjustment ensures a minimum hash length for the user before logging them in, even if crypt() succeeds somehow.
I think the conclusion that can be drawn from this is that the behavior of crypt() is howling-at-the-moon insane, and it should no longer be considered for any system built in this century, at least not without extensive wrappers that correct and normalize all these edge cases.
Conclusion is type safety is important
77 replies →
Not really. This issue is caused purely by the handling of the too-long input in the wrong way.
Firstly, why do crypt functions return a valid string to indicate an error? The problem is that the classic null return makes some programs crash which are not expecting it.
musl has a correct protocol for the error string, similar to the *0/*1 convention. It also uses two symbols, * and +. If the salt string is *, then it uses \+ specifically to thwart this issue: the program blindly assuming that the password is correct because the output of the function matches the hashed password field.
Where it seems they screwed up is this:
I.e. in this special anti-DDoS measure that checks for a too long input, they forgot to implement the key piece of logic that the output of the function, when an error is being reported, must be different from its second argument (the "salt", which is usually the entire hashed password or the contents of that field from a password file).
What we want is:
Thus the application ends up comparing "*" and "+", or "+" and "*" so the authentication fails.
Under no circumstances should "*" be returned when the second argument is "*", or "+" when it is "+", because it must be assumed that the app doesn't do any sanity checking, and blindly compares the second argument to the return value.
The conclusion is, again!, that C is an outdated tool for secrure software.
Is really the solution to a bug where a function had a wrapper that tried to solve its documented issue of returning *, to make yet another wrapper to try to solve the documented issue of returning true?
But, but 'always "succeeding" but giving an unmatchable hash is reportedly a better behavior than failing on error.'!!
Caveat:
> $ret = crypt($pw, $hash); $ret == $hash
musl is actually safe against this use. OpenSSL takes a bizarre code path, detects that $hash is too short, calls crypt($pw, $other_string), forgets that it detected that the hash was too short, and checks $hash == return value.
See my other comment.
I did note that musl would normally work just fine here. Though I appreciate that your other comment goes into much better detail about what was actually going on in the code, as my analysis basically stopped once I understood the high-level flow of what was going on.
Except OpenSSH qould have worked just fine if musl behaved correctly. Instead musl ignored the spec and tried to hack around badly written programs but ended up creating a security hole in a correctly written one!
This is a vastly clearer explanation than the linked page. Thank you!
Yep, the linked page doesn't /feel/ AI written (maybe it's using a grok or muse model whose tone I don't quite recognize yet?) but it kinda just states a lot of facts that make sense on their own but don't cohere into a whole vulnerability like the comment above does.
5 replies →
Because Claude wrote the linked page. Claude is impressively bad at communicating these days. with gpt-5.6+ approaching Claude level like 30% of time and Gemini much clearer but also the most annoying 7 year old ever, it’s a pretty lame environment especially for people who r new to everything
7 replies →
Thank you for explaining it with proper capitalization. I understood your explanation much better than the author's.
> musl can return “*” for errors
wtaf ?
There is no doubt in my mind that this was intentional.
39 replies →
One has to remember the context here. The only error possible according to the SUS is ENOSYS, literally that crypt() is not actually implemented in the library. The traditional AT&T Unix crypt() was always implemented, and its encryption did not have failure modes, so there was never any error return to handle. This is why programs were written to never check the result of crypt(). It was only ever NULL on systems where one couldn't use the mechanism at all in the first place.
So the knock-on effect is that libraries that do have errors other than ENOSYS to accommodate, have to return magic 'impossible' values instead of setting errno and returning NULL, because of the weight of existing applications that expect nothing but the standard, traditional AT&T Unix (and indeed old BSD), behaviour.
Check the GNU libc crypt() doco. It defines a whole family of 'impossible' values that signal errors, by dint of listing a load of magic characters that should never be in the encrypted string. Including asterisk.
Also check FreeBSD's OpenPAM for a case in point. Even today, it does not check for a NULL return from crypt() and leaps straight to strcmp().
* https://cgit.freebsd.org/src/tree/lib/libpam/modules/pam_uni...
If you want to see real history here, check out the 1992 comment by Eric Andrew Young, that is still in the source for OpenBSD today (in des/des_fcrypt.c), on how SSLeay's crypt() had to be adjusted so that any single character password field, including "*", did not end up successfully matching itself.
* https://cvsweb.openbsd.org/annotate/src/lib/libcrypto/des/At...
It's worth pointing out that musl tried to do the same as SSLeay, and deliberately returns something different to the original salt so that they will not compare equal. It does not return "*" if the input is actually "*" in the first place, contrary to the headlined cartoon explanation. (The locus of the problem is the caller calling it with a salt that is not "*" and then comparing it with "*".)
9 replies →
Returning a string on error honestly is defensible (especially in a language like C where the type system doesn’t have exceptions or result types), the question is why the string was “*” instead of like “crypt ERROR: input password cannot be longer than 256 characters” lol
4 replies →
> The goal from an attacker’s perspective is to get this to be true: `$ret = crypt($pw, $hash); $ret == $hash` when called by OpenSSH’s sshd.
A typo: `$ret = crypt($pw, $salt)`
weird chain of events and could have been handled in a better way for sure
"The OpenSSL adjustment ensures a minimum hash length for the user before logging them in, even if crypt() succeeds somehow."
OpenSSL??
Sorry, should have written OpenSSH.
Great explanation. This affirms my biases that shell scripting and everything-is-strings is a mistake. Horrific.
Shell is for humans to read and act on it.
Not to programmatically parse 'ls' output or some /proc/some/file.
Eg. git started as Perl scripts then moved to C to realy have data straight from C calls from system.
So for example all Internet things calling "openssl -some -options ...." are pure trash :>>
1 reply →
In-band signaling[1] strikes again. * should not have been used as special value, neither for "no password" in /etc/passwd nor "error while hashing the password" in crypt. I'm afraid we'll find many more of these abuses, specially in string manipulations. The null was ok, an exception (or equivalent) better, and Result<T> (or equivalent) best.
[1] https://en.wikipedia.org/wiki/In-band_signaling#Computer_dat...
Exceptions are almost never the right choice. They create extremely complicated control flow extremely quickly, and then you have to test all the different control flows. Java software is an example of how this goes sideways fast. There's no way to create a class that prevents callers from reading the object before the constructor runs (in multi-threaded software, new first hands a reference to the caller, which can put it on the heap, then it calls the constructor), and objects are default initialized, so all fields (even static ones that are assigned in the constructor) can be null.
So, any field de-reference in Java can produce a null pointer exception. Try testing all paths in something like:
try { foo.sideEffect1().sideEffect2()...sideEffectN(); } catch { bar.sideEffect1().sideEffect2()...sideEffectM(); } finally { quuz.sideEffect1().sideEffect2()...sideEffectP(); }
There are M * N * P paths. The only way to write testable code with exceptions is to panic on uncaught exception, and to never catch unexpected exceptions. Even then, you're relying on nothing unexpectedly throwing the exception type you caught.
I'd argue Result<T> >> NULL >> sentries >> exceptions for this reason. Sentries are terrible, but at least they're not computationally hard to use correctly.
I'd argue that's the fault of having default nullability, not the exceptions. You would have the same problem if every attribute access returned a Result<T> too.
And I'm not a fan of breaking control flow, but I still find it preferable to nulls. A null may travel far from the place where it was generated before blowing up, making troubleshooting harder, or it might come from a function you called for its side effect but forgot to check the return value. I gotta say I'm also a careful enthusiast of checked exceptions, it's great knowing which ways a function can fail.
But I think the consensus is growing that Result<T> (plus warnings for unused return values) is simply the superior choice.
1 reply →
> There's no way to create a class that prevents callers from reading the object before the constructor runs
That can only happen if you leak the reference (given we are talking about java language level semantics and not byte code), which I give you is unfortunately not that hard to do.
As for exceptions, I think they would pair beautifully with software transactional memory, which would work well with effects. In that case (and no side effects) it would not matter what code path we took, we could just go back to a known safe point.
Because ergonomy-wise I still think exceptions win (even though I do like Maybe types, and Rust's ? macro is plenty good): - they let one focus on the actual happy path code - no error is ever swallowed, unless explicitly written so (looking at you go with useless if errs that fail to properly handle the error), they are always handled somewhere - they automatically "unwrap" the Ok result, and automatically bubble up the Err side. - bubbling up is especially useful because most errors can't be handled at their source (e.g. IO error is meaningless, how you handle it depends on what you want to do in the first place, which is done several layers higher) - they automatically attach a stackframe/source, while with result types you may get a list of Ok Ok Ok Err and have no idea how that Err got there.
1 reply →
> There's no way to create a class that prevents callers from reading the object before the constructor runs (in multi-threaded software, new first hands a reference to the caller, which can put it on the heap, then it calls the constructor)
Which is why the pattern of double-checked locking doesn't work.
https://en.wikipedia.org/wiki/Double-checked_locking
3 replies →
> never catch unexpected exceptions
Yes, broad exception catch clauses are simply incorrect code.
Panic is maybe an overreaction, "scratch everything and orderly abort" is sufficient.
1 reply →
You should never ignore exceptions! The least you can do is to crash the program. Then you fix the bugs via battle hardening until it no longer crashes. Making the program not crash by ignoring the errors will just mess with the state of the program until you have no idea what caused it. It's better to let it crash, load the dump into your debugger tool to see what went wrong, then fix the bug, and repeat until you have a stable program. Rather then just live by the motto "ignorance is a bliss".
Exceptions work fine and enable a clean separation of business logic and error handling. Visibility into partially-initialized objects is a language design issue independent of exceptions. In a non-exceptional language, the corresponding access to uninitialized fields would just be bugs, not errors you can see.
Nothing in your post actually supports the idea that exceptions are "computationally hard". The CFG error edges exist however the errors are spelled
2 replies →
> There's no way to create a class that prevents callers from reading the object before the constructor runs
Doesn't this address the issue? https://openjdk.org/jeps/539
Unfortunately we’re moving to llm based systems which rely entirely on in-band signaling.
PSA: you can typically declare different types of string so that your type system objects to mixing data of different types even though deep down we know they're both strings.
If you mean type aliases, that's a good idea within a single codebase, but the distinction between different types of strings doesn't survive serialization so it wouldn't have prevented this error. What they need is a way to signal errors out-of-band, like Result<T> or exceptions.
2 replies →
This is presuming that the type system is nomitive rather than structural and there is not serialisation/deserialisation or type refinement occurring. A structural difference is much more preferable.
This is a side-effect of Unix-y "it's all plain text" design that leans extremely heavily upon in-band signalling (among other design decisions).
If /etc/shadow was a JSON (or XML, even, although that would make me feel gross), this bug would have been much easier to see, because a field that reads '"hash": ""' sticks out a lot* more than a custom microformat that you don't understand.
With data structures, there's never "plain text" - it's all structured. The question is whether you want to use existing formats and tools, or create a new one for every tool and literally make footguns like this.
A few years back I un-bashed my life as much as possible in favor of nushell (though it's not the only shell to encourage structure). I feel like it has really helped me notice a lot of this-needs-more-structure type problems outside of my shell that I otherwise wouldn't have.
The elegance of the unix-y approach is actually that it's all bits. You can just join things together and shove the bits through, that's a wild simplification compared with having vendor-specific kinds of plugs. But we could have that kind of compatibility while also benefiting from structure.
So I encourage everybody to give a shell with structured output a try, let it rewire your brain, and then ask yourself: how is the OS holding this back from being everything it could be? Because if we build that OS, it won't encourage these problems.
When I imagine it, it would probe either side of a pipe to see if they support negotiation. If not, send the bits through in the same old unixy way. If so, use the newest schema for structured data that is supported by both sides.
4 replies →
The issue isn't the config, but the fact that crypt() doesn't have any way to signal failure. In better designed languages you would have exceptions, result types, or something. Even in C you have "return integer indicating success or error" which would've prevented the confusion.
Now the public API is hard to change but you could still avoid the issue by returning a longer token like "==DANGEROUS_CRYPT_ERROR_length==" or "==DANGEROUS_CRYPT_ERROR_unsupported_alg==" which would never match the password field unless someone was really trying to induce the issue.
27 replies →
Returning “*” instead of NULL to signal failure is the real footgun. Don’t blame the format.
For historical reasons, we could’ve ended up JSON as “hash”: “*” just the same. The /etc/shadow format lets you skip values completely, like “::”, so your argument doesn’t hold.
Terminating an app because of a NULL dereference is the other footgun. Sadly, we still have “panic” in modern languages like Go and Rust.
6 replies →
Yes, absolutely. Take for example this cURL command:
Random parts of it will be interpreted by the shell, others by cURL, or the filesystem, or the remote server. They each have their own encoding rules, nested. I count ten distinct separators and nine ways of escaping/tagging. It's one command. No, just no.
We have seriously overused strings and we're now paying the price.
1 reply →
At this point I’m resigning myself to think that we’ve just gotta wait for some people to die. It feels very dramatic and glib (no pun intended lol) and whatever to say, but at this point it feels like there’s a certain hostility in some parts to changing anything from the platonic ideals of however computing happened to work in 2004. “Oh we don’t need to mess with it!”
7 replies →
> "it's all plain text" design that leans extremely heavily upon in-band signalling
Honestly I've been shaking my fist at these since 0 becomes True/Success in Bash and the opposite everywhere else
6 replies →
I'd say another UNIX-y thing - "C is good enough" - is part of the problem. C makes programming painful which results in all sorts of hacks.
E.g. in Pascal you can can define a record and then defined file of record to have typed I/O. I.e. it comes with built-in serialization format supported by compiler. Not a good one - but perhaps a step in a good direction.
OTOH C people starving for serialization often believe that `struct` byte layout made by a specific version of a compiler is a serialization format - a view not endorsed by the standard.
1 reply →
It also looked to me a bit like a semipredicate problem:
https://en.wikipedia.org/wiki/Semipredicate_problem
Even using "no_password" and "error_while_hashing" would have prevented it. The problem was picking a generic, meaningless signal string.
That would've happened to fix it, because "error_while_hashing" and "no_password" happen to not be valid password hashes. The right way would've been to have different syntax for hashes and non-hashes; you could've made it so that a hash starts with a '#' and if it doesn't start with a '#', the rest of the string should be ignored.
Or, for backwards compatibility, start non-hash strings with a character which can't appear in a hash, such as '@'. Your examples would then become '@no_password' and '@error_while_hashing'.
8 replies →
Should have set crypt_errno.
The whole story is a bit worse than this. Musl already worked around this type of failure. See the second-to-last line here:
https://git.musl-libc.org/cgit/musl/tree/src/crypt/crypt_des...
(hey, this workaround is wrong if the salt is "x"!)
but they forgot to work around it here:
https://git.musl-libc.org/cgit/musl/tree/src/crypt/crypt_sha...
The vulnerable path in musl is only a little bit vulnerable, because "*" is not actually a sha256 encrypted configuration and should not reach this code: musl's crypt(..., "*") will not call that code! Turns out that sshd detects that the password hash is too short and does not forward the salt!
https://github.com/openssh/openssh-portable/blob/master/auth...
Look, no else statement! If that if doesn't get taken, then salt remains NULL. And the sshd code is really quite messy and doesn't keep track of the fact that this code path is inappropriate for validating an existing password hash.
A workaround in the sha256 path in musl wouldn't even have prevented the bug!
I'm going to give some credit to musl for unwisely using "*" and some credit to OpenSSH for having nonsensical control and data flow.
> (hey, this workaround is wrong if the salt is "x"!)
When the salt is "x", it'll return "*" and still not match, won't it? The code is correct as long as the second argument is the hash that's being checked. And like you say, the sha512 path isn't relevant since you only get there if the salt started with "$".
> Turns out that sshd detects that the password hash is too short and does not forward the salt!
I think you're misunderstanding the intent of that check. It was added by the 2016 change linked in the writeup[1] and causes OpenSSH's xcrypt() to select and use a salt from a different line of /etc/shadow to prevent timing attacks. It is correct for a crypt() implementation that returns NULL on error.
[1] https://lists.mindrot.org/pipermail/openssh-commits/2016-Jul...
> When the salt is "x", it'll return "*" and still not match, won't it? The code is correct as long as the second argument is the hash that's being checked. And like you say, the sha512 path isn't relevant since you only get there if the salt started with "$".
I think you're right. That's what I get for doing a very very quick read.
> I think you're misunderstanding the intent of that check. It was added by the 2016 change linked in the writeup[1] and causes OpenSSH's xcrypt() to select and use a salt from a different line of /etc/shadow to prevent timing attacks. It is correct for a crypt() implementation that returns NULL on error.
I don't care what the intent of the check is. If you're doing a fake password hash to obscure timing, remember that it's fake and don't let the user log in! Don't deliberately pass nonsense into the live check and rely on the nonsense check failing to prevent the fake check from accidentally working. And don't have a code path that thinks it's checking the real password but passes in the wrong second parameter to crypt().
Or, better yet, have an entirely separate function fake_check_password() that does the fake check, returns void, and cannot possibly be confused with the actual password check code.
2 replies →
The formatting in your comment got messed btw
Thanks, fixed
My general rule of thumb is: "no magic strings". This is much easier to implement in higher-level languages and when writing application/web code than system programming.
Don't return empty string/asterisk/anything else. Instead, return a failure value.
When absolutely needed, choose the magic string with more caution:
One string showing the SHA-256 of a non-existing hash? It's a good idea for it to be a valid hash. Now we have a new risk: if it's broken once, it's broken for all. So it's a good idea for it to be a random one for each machine and/or account. Even this might have problems I can't think of, but it seems to be as secure as an account with a real password.
If this is not a good decision for some reason, choose something unmistakable: "--ACCOUNT-HAS-NO-PASSWORD--". This is much less likely to be mistakenly used in a different context, is much easier to look up in a code base than "*", and is self-documenting.
Agreed, unfortunately there is so much shit programming that does this that AI is trained on that I continually see it generating it in my code.
My AI usage involves telling it to do things and setting up lint rules to catch all the bad programming it’s learnt.
100% agreed on no magic strings
One potential point of disagreement: it's easy to implement structural prohibition of magic strings in low-level languages too if you consider Rust a low-level language (which I've heard before, even though I didn't really consider it to be one for our application of it).
I'm not a Rust expert but do run a few thousand lines of it in prod for some perf/correctness critical code, and I wish it didn't have the reputation of being so difficult to comprehend. There's so much resistance to introducing it or new languages in our large org, that I'm surprised we were able to introduce our thin slice at all. I am not sure I'd use it where you need easy async primitives like goroutines, but it prevents whole swathes of bugs that are repeated over and over again with devastating impact, like the OP's bug, via thoughtful language design.
Maybe this resistance is an artifact of where I work more than large organizations generally, though. Curious about others' experiences. I'm not really an evangelist for the language but would really like to work in a more polyglot environment one day.
Above all, I’m a little sad that the musl developers decided to make a lot of “well it’s convenient” changes to core crypto functions — at least without the level of auditing that would have caught this! It’s tarnished their reputation in my eyes a little.
I think everyone is over fixated on AI finding complex bugs beyond a typical person's understanding either to locate or exploit.
I think for any one of those bugs there might simply be half a dozen footguns or bugs that once it's spelled out you wonder how anyone let it slide.
I think that's the case where LLMs might indentify most.
It's challenging though as the current AI audits produce so many false positives it's hard to know when it's being pedantic or when it's genuinely caught a human being a bone head
Finding more and more that these complex bugs people gloss over like you are now are being used in exploit chains and the end result is still my clients getting hacked
This one might be hard to find using the author’s approach of throwing everything at various builds of ssh.
It is not really that hard to find in the wild if you had a device built this way, when overflowing buffers on embedded devices is a standard attack vector.
It might be hard to root cause, but it would not be that hard to find.
Honestly the ideas around musl are interesting, especially being able to statically link, but given the sort of nonsense around how they handled pthreads, and generally other trade offs, it's hard to imagine wanting to use it over glibc (which everyone shits on) if you don't absolutely need static linking.
Everything in this industry is about convenience.
Musl has a history of introducing (and not fixing) major bugs in their library that aren't in glibc; I would not depend on it for production
In many ways each individual decision makes some sense; wanting to stop loads of programs being affected by a (remotely triggerable?) crash wanting to avoid time based attacks wanting to be able to set accounts to not have a password
Each one relies on a perfectly valid argument - that * can't match because we all know it'll be rejected by the other thing.
I would be more concerned about the lack of explicit return types, one of the arguments for strongly typed languages is that you should not have ambiguous data and yet…
Granted I mostly work with high level programming languages so I may be missing something.
1 reply →
> Each one relies on a perfectly valid argument - that * can't match because we all know it'll be rejected by the other thing.
But that's not a perfectly valid argument, and it never was.
I guarantee that every single developer of musl in 2012 had at least account on their machine with "*" as the password hash. Also crypt() is a standardized function and the standard states that a null pointer is returned on error.
That being said, OpenSSH did mismanage their attempts at handling a garbage return-value from crypt(), so there's definitely some blame to go around.
If they had just all used a different garbage symbol would this have been avoided?
The real problem here is the C language. Our industry should have dumped it a long time ago and we’re now paying the price of prevarication.
53 replies →
You must be new here. While I applauded the idea of musl when it first came out (coincidentally, glibc wasn't having the best time in its development then), the developer's arguments why <some behavior drastically different from every libc under the sun> was akchyually correct, which was why you had to fix a program that hadn't been broken for 30 years and now suddenly was, made me put him under the "clowns" category and never look at musl seriously again.
When later it turned out that musl's procrustian stack size made Python not really runnable under it, it cemented my resolve to not go near musl.
And now we have this.
I’m also reminded of the clowns pushing “it’s smaller!” container images based on social media vibes rather than analysis where, yes, not using Debian+glibc saved tens of megabytes on the base OS but did so at the expense of measurable performance reductions & compatibility issues like the one you mentioned, and literally every single one of those dudes I worked with was shipping containers pushing a gigabyte so the size reduction had no measurable impact on deployments.
Uh, isn't this the case of the opposite? The musl devs trying to accomodate broken programs?
I am sorry, a little? I am deeply disappointed that libc decided to do the same just with another sequence.
And why because some dude was complaining sshd or whatever was crashing because it didn't handle nulls? Just handle the null my guy you already know why your bullshit is crashing.
Let's be real here, the main culprit is C. People love to say "but you can still get logic errors with Rust", and no shit of course you can. But it's much less likely. You wouldn't have gotten this bug.
A little?!? I’m sorry, but musl is a joke, this is yet another proof of that fact, and I have no idea why anyone uses it in production, they have only themselves to blame after the many warning signs.
In my last org I was in charge of internal testing and qualifying software releases for network equipment, and designing test harnesses and tests for these. Significant install base. All the major vendors. Many years. Basically doing the vendor QA for them, because we couldn't afford things to fail. As you do - I wish.
This was not security testing as such - mostly latency, performance and hardcore network "can we break it" and "does it comply" stuff, but also a lot of cable and module / line card pulls, pluggable and fiber breaks, random reboots and pre-empting a lot of possible operator inventiveness. At one point we considered SEU bit flip testing on rented cyclotron time.
And always we battled against the top brass urging us to speed up the process and don't bother with "really basic" testing, because we observed software quality going down over the years as the push for release and time to market stared to dominate.
So we always started with sanity testing - and that included oversized password and CLI input strings. Always. And we caught things more than once, but never like this.
We were right after all.
> We were right after all.
Exhilarating, isn't it?
Hardly. I say this with a faint, bored "huh" note rather than with a triumphant "I told you so", and I move on.
It's about maintaining solid engineering practices and far above proving who's right or wrong.
This is why I don’t fully buy the argument that software that’s been around for a long time is more secure and reliable because it’s been battle-tested. There’s been a lot of time for cruft to accumulate until no-one can ever fix it because everyone is expecting it to function in the broken manner it’s always functioned. And some of the more obscure configuration options get a lot less scrutiny.
Musl, which is vulnerable, is much younger and less battle tested than glibc, which is not vulnerable though
This is not an obscure configuration option. Your password file almost certainly has at least one account with "*" as the password hash.
Integration of different software is the stage most likely to introduce uncaught bugs. That is what happened here: musl intentionally broke the documented API of a function used by OpenSSH. Then Alpine integrated this incorrect implementation with OpenSSH.
Fun!
I was able to reproduce this on Alpine 3.23.7 with OpenSSH_10.2p1.
Luckily PermitRootLogin [0] hasn't allowed logins with passwords for root since 2015 [1]. I'm guessing that's at least part of why this went undiscovered for so long.
0: https://man.openbsd.org/sshd_config#PermitRootLogin
1: https://cvsweb.openbsd.org/diff/src/usr.bin/ssh/sshd_config?...
Did anyone get it to work? I tried on the official alpine docker image, enabled password login for root. The shadow entry has ! as password by default, this didn't work with a > 256 character password (i also did passwd -l which also does !). I manually changed it to * and it still didn't work.
Am I doing anything wrong here?
OpenSSH_10.3p1, OpenSSL 3.5.9 29 Sep 2026
PRETTY_NAME="Alpine Linux v3.24"
It's because alpine has updated all openssh-server packages already. Even from EOL alpine versions.
Ahh! Wasn't working on 3.16, which does indeed have an updated package.
But, got it to work on 3.15 - which requires changing the ! to a * in the shadow file in addition to other steps (making another user with a hash, setting PermitRootLogin to yes, etc.).
https://pkgs.alpinelinux.org/package/v3.16/main/x86_64/opens... Build time 2026-11-05 06:53:46
1 reply →
oooh, that'll do it.
Didn't notice that the openssh-server-10.3_p1-r2 was released today.
EDIT: works after downgrading OpenSSH, turns out even EOL Alpine version got updated packages!
Also didn't have much luck on Alpine 3.20.10 with OpenSSH_9.7p1 which I've freshly installed from an official ISO.
Last two entries in /etc/shadow:
SSHing into test@ with one 'wrong' password then one 'long' password doesn't work. I have UsePAM no and PasswordAuthentication yes in my sshd_config.
No, didn't work for me open OpenWRT x64 running in a container but that is probably 'cause I'm using dropbear instead of openssh-server. That's what OpenWRT uses by default so the impact there will be minimal assuming that most installations don't change to openssh-server - it isn't installed by default.
On top of OpenWRT on my router (running 25.12) running dropbear instead of openssh-server, I also do not have any accounts in /etc/passwd|/etc/shadow with "*" for a password hash and an actual shell (all are set to /bin/false).
Nope, wasn’t able to reproduced it in my firecracker VM, which isn’t publicly accessible anyway.
As far as I understand, it wont work on default setups. You need musl-c, no pam and openssh.
In Alpine, these would all be the case for an out of the box install with openssh added unless you specifically install openssh-server-pam and configure sshd_config to use PAM (Alpine is already musl-c based and doesn't default to PAM).
3 replies →
In Unixese, shadow having ! means the account is "locked", right? I haven't tried the steps to replicate so maybe they did replace the * with a ! or something, but I would try making sure the account is not locked.
Did you try one incorrect standard password fail first?
I can't reproduce it either. I used alpine:3.24.1 and openssh is openssh-server-10.3_p1-r2
I tried adding PermitRootLogin yes and PasswordAuthentication yes. Tried adding another user with a valid password.
Verified shadow entry has a *. Can't login as root or a test user. Tried another incorrect password first as well.
edit: can reproduce after `git revert 384e25e6bd1f2e7b30095900e48b570d46953510` in aports and rebuilding openssh without the fix. It also accepts the password on the 2nd+ attempt, NOT the first.
[dead]
[dead]
Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option because it means people actually notice that there's a problem. Silent compatibility hacks are the worst, they guarantee problems never get fixed and the stack of hacks never gets smaller.
Microsoft has historically been one of the leaders on the wrong side of this one, but they have also demonstrated how effective it is to call out badly behaved software and make it annoy the users with UAC. When Vista came out, so many applications needed elevated privileges to run because they were badly designed to write user data in system locations, but now their application was more annoying to use and came with a scary dialog about it making changes to your computer that other better-designed software didn't get.
More often than not if a program didn't actually intend to change system settings it was just a matter of badly chosen default paths which are pretty trivial to fix, but so many vendors just would not do it despite years of complaints from NT admins wanting to not have to spend hours with Process Explorer figuring out specific permissions to grant limited users enough access. Once their failures were put front and center, calling their application out as the one getting things wrong every time the user ran it, they suddenly had motivation to fix it.
For security it's better, but in many cases if you're not writing security-oriented software it can be worse from a user perspective. I remember the first job I had over 20 years ago was working on VoD streaming and one of the "tricks" to improve stability was mmap'ing the 0 page so that NULL dereferences wouldn't immediately segfault. Ugly as hell but when you were working with all sorts of more or less well coded media codecs and security wasn't a concern at all (worst case scenario the user would stream movies without paying) it made some amount of sense.
I suspect that Microsoft has the same history: the entire Windows platform was built with single-user PCs in mind, mostly only dealing with trusted input. The users were also expected to be tech-illiterate for the most part and unable to troubleshoot deep technical issues.
More recently I'm still working on video transmission, but on the production side this time. I also originally adopted the mentality that I should crash early if some unexpected situation was encountered, but I ended up walking that back a lot, because having a critical piece of live media production reboot mid FIFA worldcup final is generally Not Great. Degraded functionality is usually preferable. I still crash early in dev builds, but the production code just logs an error and continues if at all possible.
That said, I definitely agree with you for something like password handling. In this case, I do think it's warranted to crash the program without survivors if something is amiss.
> if you're not writing security-oriented software
It's hard to define what's security oriented and what is not. For example, is this HN web app security oriented? Not in the way of sshd but it must keep our accounts data separated and it must not let people exfiltrate data from the db, where for example it is storing the emails of some of us.
1 reply →
> because having a critical piece of live media production reboot mid
It's not great, but these days a trickster with AI will find some way to start broadcasting porn to a billion people because of the flaw, so may end up being the better option.
1 reply →
> because having a critical piece of live media production reboot mid FIFA worldcup final is generally Not Great.
Story time?
1 reply →
> Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option because it means people actually notice that there's a problem.
100%
Agreed. I feel like this attitude has been holding Python back, too. It's one thing to criticize "move fast and break things", but it's also not good to ignore already existing breakage just because the pieces still fit together neatly, and it's not good to refuse to move at all.
> letting badly programmed software die when unexpected things happen
Go ahead and program a flight controller this way, I'm sure nothing bad will happen.
I'd still rather have the software crash and alert a human that to continue in a faulty state. The way to add robustness in this case is to add backup systems, not "ON ERROR RESUME NEXT".
1 reply →
> badly programmed software
Is there a test or check I can use to determine that software is 'good'?
Oh there isn't? Good/bad is subjective? We need to scape goating hacks as bad software, as if anyone is capable of perfect code. Perfect code would require knowing the system inside and out, the stack up and down - no one knows all that, no one is perfect. Sufficiently advanced AI though can rip through all of that, find the weakest spot and zero day it, bringing the entire house of cards down.
Software you thought was good can still be hacked. There is no way to create provably unhackable software.
Which means all software is packed with vulnerabilities just waiting to be discovered, no matter how good you think it is.
Now everything installs to a user local folder, runs as the user, there are no UAC prompts, and malware can just freely tamper with other applications, act as the user, and harvest all data the user has access to.
Hey, but at least it can’t touch the System32 folder.
Great job guys.
> Microsoft has historically been one of the leaders on the wrong side of this one
Objective-C ahem nil messaging...
Objective-C is an Apple thingie :)
3 replies →
> Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option
Sadly, we will never learn this because failing fast runs counter to intuition, and a class of person will never override it.
[flagged]
So to recap, with a non-upgraded sshd you need:
- an account with /etc/shadow hash set to '*'
- (note root won't work by default because of PermitRootLogin prohibit-password default)
- attempt twice, first attempt fails second long password is accepted (that took me a while to understand)
In practice passwd -l seems to lock with a ! prefix, and new account without passwords are created with a single ! which is also not affected, I'm not sure how big the impact actually will be in practice and don't understand the recommendation to upgrade urgently right now... Just don't use `*`?
EDIT: ah, some default container images apparently have a default alpine user with that hash, so I guess that's bad for them. This makes a bit more sense now.
> some default container images apparently have a default alpine user with that hash
Do you have an example? Looking at Alpine's advisory, I can only see that their cloud images (i.e. what you use to run a VM, not a container) have a default alpine user, no container images. And even with those, cloud images with cloud-init are unaffected with default config due to `PasswordAuthentication no` being inserted by cloud-init into sshd_config. So that leaves... Cloud images that use Tiny Cloud for provisioning. That's an instance bootstrapper authored by Alpine from the looks of it so I'm not sure how many people would prefer that over cloud-init (which a lot of VM/bare metal management software may already make easy to use) or just a non-cloud image.
Why do you need to attempt twice?
> Just don't use `*`
Well but, alas, a non-zero number of user probably do use it, though and "upgrade X" is a clearer and better instruction to communicate than "change this setting where maybe you don't even know what it means and it might break your workflow, but saves you an upgrade".
In some contexts (e.g. embedded) you'd be surprised at how reluctant some people are to upgrade when they can run through a dozen of hoops for workarounds such as changing hash instead...
It's amazing how similar this exploit payload is to the 20+ year-old Quake 3 vulnerabilities found by Luigi Auriemma (aluigi), which also used the exact same 256-to-1024 character range and nothing more than a long string of AAAA.... Here you're feeding >256 A's up to a 1024-byte limit to trip musl's defensive length cap, whereas back in the mid-2000s, Luigi was famous for publishing one-line .cfg scripts where the entire exploit was literally "say aaaaa..." repeated a few hundred times overflowing server's memory.¹ ²
A great example is Jedi Academy, a 2003 Quake 3 engine game whose multiplayer is still actively played today (its open-source engine port, openjk, is even in the Debian repos btw). Luigi had two classic scripts for it on his site: Jedi Academy <= 1.011 in-game server buffer-overflow (jamsgbof)¹, which crashed the server just by pushing a string of as past the 1024-byte console buffer, and Quake 3 engine in-game players kicking (q3msgboom)², where a wall of as hit the engine's 1022-char truncation limit and the leftover bytes confused the parser into disconnecting everyone on the server.
Kind of wild that 20 years later, the exact same payload goes from crashing Quake 3 matches from the tilde console to getting root on some SSH servers.
[1] https://web.archive.org/web/20101227181948/http://aluigi.alt... [2] https://web.archive.org/web/20060522175643/http://aluigi.alt...
Offtopic: this is a very cool way to present commits and code excerpts. Love classic Windows styling.
Were those pictures with links generated using a publicly available tool that we could use, or were it custom (vibe)coded specifically to create this page?
This looks entirely vibe-coded to me. No one is going to put so much effort for a vuln.
To be honest, I love this sort of vibe-coding. Explains it really well, in an interesting way. If this is vibe-coding, then the AI has done a great job.
2 replies →
Yes but it's not your average vibe coded site, it's still decently done and clearly art-directed by a human rather than "make cool website for me plz"
MITRE scored this as a 8.1 with a high attack complexity. [0]
I fail to see how the attack complexity of this vulnerability is high, as all it requires is attempting an ssh login with a password longer than 256 characters to an affected alpine image with port 22 open. Am I in the wrong?
[0] https://nvd.nist.gov/vuln/detail/cve-2026-123158
From the advisory: musl’s crypt() returns the string * as an error indicator when the key is over 256 bytes long
Are these people mad? Why on Earth would anybody indicate error like this?
That's a well-made site, yet I feel nothing.
I thought it was a great write up. It did feel a bit AI-coded, but whether it actually was or not, it's still a good bit of work imo.
I dunno, there were some strange discrepancies in the explanation (and captions). Didn't really hang together.
Looks good overall, but I was wondering if eg. "the timer learns nothing" and "no X, no Y, no Z" was slop rather than a deliberate choice.
1 reply →
fwiw pangram believes the text is AI as well
It would be weird to spend this much effort doing the site by hand, for a bug that the author says was found by a swarm of agents.
LLM output fatigue?
Apple sideways traffic light fatigue.
2 replies →
I didn't hate the writing even if it's AI, but I deeply disliked the animations. I don't know if that's LLM fatigue or not.
I left after trying to scroll to a concise description of the issue. I don't want a theatrical presentation.
What is dead can never die. All blueteamers are ironborn these days.
Maybe that's on you though. I enjoyed it, though the discrepancy between the `*sshole` name and the `sshhole.com` domain made me twitch. How is anyone[0] obsessive enough to find this and write it up and yet not obsessive enough to make the amount of `h` consistent?
[0]: Be they human or LLM. Or an entity with any proportion of the two. I do not judge nor discriminate.
Look closely. The second h is behind the word and almost as dark as the background
2 replies →
It has very low information density. This would read 10x if it were just a few paragraphs of text. Instead we need to render an animation every few sentences. Not a fan.
It has a crap load of text but says very little and is very bad at conveying the details because it thinks it has to tell some weird story.
I had this classmate that had their ChatGPT instructions to respond in this same laidback, lowercase, gen-z style. i recognized the output as a lot of llm writing almost immediately
1 reply →
I do actually. My guess is that this was not just one-prompted, but someone took the time to refine it based on their tastes and expectations until it matched something they would actually like.
However, if someone already developed a strong AI-slop-tingle it will still be triggered by some telltale signs and let their subconsciousness know that they can't possible enjoy this.
I think at some point I'm going to get banned from HN for repeatedly saying this, but here we go:
This thread highlights why I often say that at some point in the future, the obsession with whether something was written by AI or not will be classified as a mental disorder.
It is tiring to virtually every day see this discussion play out in some HN thread or other. It's also just noise. I don't think your comment, nor the thread it inspired, added anything of value to the discussion.
For OpenWRT, AsusWRT etc, I believe they use dropbear ssh by default instead of OpenSSH, which apparently isn't affected?
Yes, correct for OpenWrt. OpenWrt defaults to dropbear for both server and client. However, more advanced users may optionally choose to replace it with OpenSSH, or install OpenSSH side-by-side, to gain additional features of OpenSSH.
I've just tried with my AsusWRT (3.0.0.4.386_69196) and didn't get it to work
gregor@krekrm3 ~ % ssh root@192.168.1.1 root@192.168.1.1's password: Permission denied, please try again. root@192.168.1.1's password: Permission denied, please try again. root@192.168.1.1's password: Connection closed by 192.168.1.1 port 22
Dropbear added the 100 character password limit in 2019.77, so earlier versions might be susceptible.
Nope, they are OK.
Wait so musl, famously a stickler for standards, just changed the behaviour of a POSIX function?
To be fair, I have less than zero sympathy for anyone who runs SSH on any platform without `PasswordAuthentication=no` in 2026.
Its not asking for trouble, it is there will be trouble, not if but when.
As for the people who say "well, I run SSH on port 942329053052". Well guess what, the bots make short work on that. We tried it at $work as an experiment, multiple bots found it in less than a couple of hours. That's why your SSH config must be hardened, preferably running on a hardened OS too.
> Its not asking for trouble, it is there will be trouble, not if but when.
Did any of those bots brute force a password? Because a 20 character random password is comparable in brute force complexity to an SSH key. The experiment you ran was just as valid for SSH keys on that port, and had no bearing on password vs key.
> Did any of those bots brute force a password?
The point is they found the security by obscurity so-called "hidden" port in less than a couple of hours.
At that point, what they did is irrelevant. The point is that security by obscurity does not work in 2026.
> Because a 20 character random password is comparable in brute force complexity to an SSH key.
You can't be serious with your comment.
No its not.
And even then, SSH key is asymmetric, you're not sending the private key to the server. But with a password you are, which leaves you vulnerable to many attack vectors that simply do not exist with keys.
5 replies →
How is password auth weaker than key auth? Bugs in auth code can happen in both pathways. And apart from stupidly short passwords both should be equally impossible to brute-force.
In other words, conceptually, at the end if the day, isn’t a key just another name for a password, really?
Password gets sent to the server. Private key does not get sent to the server, and authentication protocol can be forwarded safely, enabling things like ssh-agent, FIDO SSH keys. This is solvable with passwords (see: augmented PAKE), but `PasswordAuthentication` isn’t that. But there are also settings where none of this really matters, yes, to be fair if that’s what you’re referring to.
SSH certificates are cool too.
3 replies →
What people think of as a very strong password often has <100 "security bits" of entropy.
Keys are usually at least 128 bits of entropy, and it's easy to take them up to 256.
Yes, both are a shared secret, but password auth - even with challenge-response - is weaker than key auth exactly because key auth provides more guess resistance.
15 replies →
Password reuse is the problem. The password hash is stolen from some other site and brute forced offline.
2 replies →
> How is password auth weaker than key auth?
Well, it's a heck of a lot weaker in this instance (at least when `PasswordAuthentication no`)
1 reply →
you claimed my favorite game's character name!
100% and with local tool calls many AI systems will do ssh "for you" and end run around any security theatre that you may have going on with LLMs.
I run SSH with password authentication on but the port binds to a ULA address. It’s fine. No need to be so absolutist.
> No need to be so absolutist.
Why not ?
Even if you don't want to go all the way to SSH certificates, SSH public-key authentication is laughably easy to do.
I also don't buy the argument "but its on a ULA address". Attacker's old favourite these days is finding a way in (North–South) and then using that as a stepping stone (East–West). Also the general concept of "lower security is OK because its on the LAN" is so 1997.
4 replies →
Running on that port would be perfectly safe as it's outside the range of valid port numbers.
This particular bug works even with password authentication turned off.
Where did you get that idea? The entire bug is about password hashing. With password auth off, that's not even attempted.
1 reply →
Bug > are password logins still on? they are by default
You > This particular bug works even with password authentication turned off.
No. The bug is related to password authentication. If you do not have password authentication then you are not affected.
What freaks some people out is they chose parameters where they do have password authentication but they expect to always fail. That's a bad choice, like OK you did fit a cheap manual keyhole, but you threw the keys away thinking now nobody can open that - you realise somebody with a cheap jiggler can open it right ?
1 reply →
> This particular bug works even with password authentication turned off.
Does it ?
Somebody here has already said "You can set PasswordAuthentication=no or update to remediate"
We don't use musl at $work anyway, so I've no means to test.
N=2 but non-22 ssh ports get almost zero traffic for me at home and at work. Not zero - but very nearly zero. shrug.
I mean there are 65k possible ports so it's the equivalent of guessing an 16 bit password, i.e. two (extended) ascii characters. It's not very surprising that 'multiple bots' found it: a single `nmap` command will give you the open ports in a few seconds.
What is out there that uses musl without PAM and hosts an OpenSSH SSHd with PasswordAuthentication?
As in: "Actually out there"
The hypothetical Alpine exists, yes, but what runs that like that?
What is out there that uses musl without PAM and hosts an OpenSSH SSHd with PasswordAuthentication?
Possibly public SFTP servers. I have allowed null passwords in the past for anonymous public SFTP and I run Alpine Linux exclusively for my servers both VM's and physical. The shadow package (PAM) did not get installed by default. I installed it but I had to specify it in the installation so there are likely people without it. There is probably a way to find them on shodan.io.
I will activate one for people to play around with for today:
There's no firewall and no password. Hack away HN.
cd /pub
get so*
Hopefully some had a chance to have some fun. I moved sshd back behind haproxy. There's still no firewall but ssh should be invisible now.
A large number of Docker containers?
Edit: Sure, a small minority presumably run sshd under these conditions. But a small minority times the number of Alpine containers running in the world gets to a large absolute number.
Edit 2: You really don't need to look very hard to find public container examples at million-download scale.
Many, many Docker containers rely on "docker exec" or the kubectl equivalent to gain CLI access when needed. To have sshd in an Alpine Docker container, you're generally need to be running init and a bunch of demons, not just a single standalone service.
I'm guessing in practice relatively few people are actually exposed. But if they are, it's catastrophic.
Docker containers run sshd?
> Edit 2: You really don't need to look very hard to find public container examples at million-download scale.
It would be super helpful if you would answer a "what, specifically" question with a "this, specifically" itemized list or not answer at all.
This is not helpful.
Docker containers that run sshds?
Agent sandboxes today probably
Its kind of cool to see such simple to follow logic bugs instead of usual memory safety issues. Each step made sense in isolation.
The terrifying decision for the API to return "*" on failure is a second-order consequence of memory unsafety, because it was a half-baked attempt to mitigate a common source of UB that people were encountering in the wild.
i'd say it is more a consequence of not having exceptions or other way of being explicit about error values.
1 reply →
From the other thread’s source [1], it affects musl without PAM. You can set PasswordAuthentication=no or update to remediate.
[1] https://hachyderm.io/@damienmiller/117563026759098135
Wow. I wonder which of the common fuzzers would have found this a decade ago. If literally any password over 256 chars works, that means nobody has ever fuzzed openssh on musl linux
In the TXR Lisp run-time, I spent a disproportionate amount of time on the crypt wrapper, making sure that it detects errors on numerous version of crypt, and converts them to an exception.
I settled on this. There is a block comment on the error handling, dating back to just 2023. There is a tiny infelicity in the wrapper in that we rely on the value of errno in the error path, but we do not set errno to zero and check for it being nonzero (evidence that the function set the value).
Applications using crypt directly can do the same thing: if it returns null, that's an error, of course, but don't use the result if it is less than 13 characters: that's also some kind of error token. The classic crypt() puts out a 13 character hash. Everything else is longer.
This is an unpleasant era to live through, but at least on the other side we may actually have real tools for writing secure software.
This kind of thing is a constant worry of mine, but I've never had the (mis?)fortune to look at this particular code.
Signatures/verification (typically) only provide a boolean guard on what you want to be controlled behavior.
To properly protect against cla controlled behavior, the check must be incorporated into the EXECUTION of the behavior (with a safe fail).
This is hard and I think a lot of pentesters will continue to have lucrative work hunting issues like this down for a long time.
Fundamentally, the problem is the "type" of the verification and the execution.
The musl vulnerability looks a lot like an intentional backdoor, doesn’t it?
Choosing * instead of NULL is indeed very fishy coincidence. Of all the return values they could've picked... What are the odds?
About 1 in 256 I'd reckon
Also the coincidence of 3 weeks later adding "oh, no passwords above 256 chars" making it easy to trigger
At this point it feels to me we aren’t discovering security bugs but a decade of planted backdoors.
Never attribute to malice that which yada yada yada
And like who even needed to plant back doors? If you're the NSA you already had an agent swarm in 1993, but it was made out of dudes from Berkeley and New Haven instead of racks in Prince William County and 印西 (and they were literal secret agents I guess, pretty metal). You can get back doors just by foraging, no need for agriculture.
No the biggest lesson of all this AI stuff to me is that we are just not smart enough as programmers
If you think about it that’s the whole idea of neural nets
We aren’t smart enough to design the function we want by hand as programmers so we need to learn it through weights
This lesson just keeps recurring in many different ways
Be careful, saying this gets you downvoted like https://news.ycombinator.com/item?id=50364646
actually this looks very underhanded indeed, what are the odds they pick the same sentinel character?!
Like 1 in 5? The set of go-to characters for that kind of thing is pretty small.
1 reply →
Pretty big? For this purpose, my instinct would be to select an ASCII character that's not alphanumeric, not directional, and not a common separator, and that's like, what, a set of 11 characters?
You have considerably underestimated both how fallible humans are, and how high their tolerance is for risks which aren't immediately visible.
This isn't even just the software industry, humans are like this about everything. Imagine explaining to something rational that (a) humans didn't realise that they won't be able to competently operate a motor vehicle while watching TokTok dances, and (b) when we finally made that illegal because it's such an obviously bad idea they would keep doing it anyway.
This is why those "What if it builds SkyNet?" conspiracies were so stupid. Why on Earth would you build SkyNet when you can just invent Weeaboo and the idiot humans will die playing ?
True, but also, that is exactly the pretext that would allow someone to maliciously, intentionally commit a backdoor.
There was even a contest for hiding/obfuscating malicious code, e.g. as an honest mistake: https://www.underhanded-c.org/ . And what do you know, the most recent, common approach shown on the page, is: improper comparison inputs that give unexpected comparison results, which is... well... this submission.
1 reply →
[flagged]
I expected let another AI slope website and instead that was a real human writing about what they found. Sure they used LLMs to find this, but they used their real fingers to write this. See, this isn’t that difficult, and it’s a lot more enjoyable to read.
It's definitely an AI slop website even if a human wrote the text. I found it pretty hard to get through the garbage animations.
I don't think a human wrote the text either, it's likely edited after generation: https://paste.c-net.org/ExtortSincere.
> mostly the boring parts where it talks to libc, like the password code
Is clearly not human.
2 replies →
It's just a LLM instructed to write everything lowercase. Fooled you
I think the following should be done. Either this, or eventually DataKrash will happen. International coordinated emergency should be declared with an immediate order to shut down all non-critical internet exposed services. Critical services should be put under armed guard and remote access should be disabled except for whitelisted critical staff. Then lustration of programmers should begin. Everyone who ever coded in C(rime) language and any descendant languages would be barred from this profession at least for next 30 years. Then maybe new software will be written with due care, and any new services should be put online only when written without any C - no OS code in C, no library code in C, no C conventions, etc
I avoided using alpine-based containers ever since finding out a lot of Linux server software is actually Linux + SystemD software. The union set API of both Linux and SystemD seems to be the default "Linux API" today.
Container images rarely ship with an init service of any kind (systemd or otherwise); the expectation is that your application is the root process. This includes Alpine-based images.
but alpine famously does not use systemd but openrc?
That's their point. The software they want to use often assumes its being run on a platform with systemd, so running that software on Alpine which makes a point to not use systemd causes challenges for them compared to using a different distro.
I use alpine-based containers whenever I can just to keep the world safer from SystemD rotting Linux.
Never enabled sshd there... except on my PostmarketOS PineBook Pro... oh shit!
What is the argument against systemd?
Is it mainly that it is too big?
I feel like everybody still loves the unix philosophy, but the previous tools were too bad, systemd just filled user demand for easier tools.
The moment someone makes a pid 1 software that is good and still modular, open and pluggable I think that will be an instant success.
3 replies →
Why is this not yet published on the official website and why is there no CVE? https://www.openssh.org/security.html
Edit: My bad as pointed below this is not in OpenSSH but in musl and it appears in the relase notes https://www.openssh.org/releasenotes.html#10.6p2. I had scrolled straight to the bottom of the website because the animations annoyed me. Still would like a CVE for proper tracking though!
OpenSSH considers this to be a vulnerability in musl and not OpenSSH, but shipped a fix anyways since that helps mitigate impact... Whether you agree with that or not, it's up to you :)
Probably because mitre take days to weeks to give out CVE numbers (if ever). Just get the fix out there.
The last few weeks have made me feel very good about my policy of zero internet-accessible ssh ports, ever. On AWS, use SSM. For a VPS or home server, access everything over WireGuard.
Such solutions always felt like swapping one risk for another. With good separation (running the VPN on a separate host from the SSH login, otherwise a bug in the VPN solution still might result in a shell on the box), at best you'd need two vulnerabilities before someone gets in, but it's not fundamentally different unless you find a way simpler protocol where you can be more certain that it's secure than ssh is (for all its scrutiny and track record of no pre-auth bugs for decades)
Putting services behind ssh forwarding was the way to secure something you're worried about rather than the other way around, but Wireguard actually makes a good argument nowadays. Iirc they did proofs for the cryptography and it has good design in terms of attack surface, and it's old/mature enough now that I'll say it's probably better than exposing SSH. On the other hand, when this thing running in your kernel gets compromised, you're slightly worse off than with someone logging into a passwordless service account as in the current bug
One thing that makes me feel a lot more confident about exposing a WireGuard port is it’s completely invisible unless you have the host’s public key. Nothing will show up on a port scan.
Its security model doesn’t rely on that fact of course, but it’s the cherry on top.
Of all the named vulnerabilities, I appreciate the history lesson and screenshots of how it got there
Thirty-odd years ago, you could get to a Unix shell from user terminal session of the Vancouver Public Library (British Columbia, Canada) by entering a super long line.
From there, even though you were not root, perms on important stuff were wide open: rxwrwxrwx.
the more i read about this, the more i'm convinced, contrary to what seems to be the prevailing opinion here, that this is primarily an OpenSSH logic bug.
which is not to say that crypt() is well designed or specified - it's clearly deficient. but I can't fathom how the OpenSSH maintainers could come to the conclusion that the root cause is "outside OpenSSH itself" when the root cause is that OpenSSH's mitigation of timing attacks rested on faulty assumptions about the return value of crypt().
A common complaint I've seen (and I experience on occasion) is AI agents being a bit silly in their overprotective coding style. They nit pick and catch every single little error. This can be frustrating since a lot of the time they are trying to handle edge cases that are so unlikely to happen, yet to cover them requires significant additional code.
This might be the beginning of the reason we shouldn't be reading the code. It's like in chess engines where the players complain about how weird, alien and unintuitive the moves are. But 100 steps down the line the engine finds some edge case exploit that gives it enough advantage to win.
So we may be crossing a line, right now, where the non-intuitive alien code being generated by agents is seeing edge cases down the line that might allow exploits. And if we "correct" the agent, we may be leaving ourselves open to exploits we don't see and have difficulty understanding.
The answer is not to reduce our understanding. That is the worst possible conclusion we could draw from what we're seeing. The answer is to increase it.
That is similar to saying to Magnus Carlson that he should just increase his understanding of chess.
The point of my comment is, no amount of "increased understanding" puts a human on equal footing to a chess engine.
> non-intuitive alien code being generated by agents
You're saying it as if humans can't write that too.
I am saying that we won't be able to write it, in the same way that Magnus Carlson (or whoever your top chess player would be) cannot compete with a chess engine.
I am arguing that, much like the "best" chess move is unintelligible to best chess players, truly secure code will be unintelligible to humans.
1 reply →
Tangent: Pangram says the text is 60-ish% AI-generated. Not that surprising given the look of the page and the work being done with AI in the first place, but it is interesting to me because this doesn't look like your standard wall-of-text AI purple prose. So it's notable that a) you can generate convincing text in this style and b) Pangram detects it anyway.
Panagram is just guessing.
PSA for OpenWrt users: If you are getting the post quantum message on login which points to openssh.com, you are likely using dropbear nonetheless. Check in System->Software. I was confused and worried why I seemingly had the vulnerable OpenSSH installed, but its just dropbear using an OpenSSH link.
So AFAIU, you are fine.
The bugs and commits that "fix" them reveal somewhat of a shitshow for how musl is managed. All projects have bugs, but I can't help but question why the wheel was reinvented here. Many toes have been stubbed along the way.
Nice web site. However, I've got a server with 14 alpine-based docker containers in production. Among them 13 are widely distributed images (like web server, databases, etc.) and one is a custom build image. Only one of them have an SSH server publicly exposed, Forgejo, but it's its own internal SSH server which only accept public keys and Git commands. On the internal network, only 3 of respond to SSH. I haven't tweaked their default SSH config. None were vulnerable to this attack.
I tested an unpatched forgejo (enabling passwords temporarily, and even adding root to permitted users) and couldn't trigger this.
ETA: I could trigger it. You need to enable passwords and add a second user with a valid password, then git@ works. Obviously this still means the default forgejo is safe as you say.
A+ name
A* name
A{256,} name
2 replies →
Did we declare "break the world Thursday" and nobody told me? I feel like it's three weeks in a row that we've had a once-in-a-decade security issue pop up.
I think we should get used to eternal "break the world Thursday", just like our predecessors had to come to terms with Eternal September [0].
[0] https://en.wikipedia.org/wiki/Eternal_September
I have alpine linux on an arm device, and I have all the prerequisites for the exploit, yet I am unable to exploit myself. All of the Who is affected? from alpines security advisory applies to me. What am I missing?
Update: I think I got the length of the password to enter wrong. I successfully exploited myself.
it's like the reverse of the Max Levchin story about almost losing the master key for PayPal's payment credentials (not gonna spoil further than that)
https://max.levch.in/post/724289457144070144/shamir-secret-s...
Microsoft Teams has flagged the sshhole.com as a malicious domain and now there is a org wide investigation where I work happening right now lol
Magic characters are awesome.
> except a lot of programs never check for NULL. they just compare whatever crypt() gives back with the stored field, hit the NULL, and crash
> so musl (the libc under alpine) goes ok fine, crypt() just won't return NULL anymore. if it fails, it returns a single character instead.
Considering it's been accepted that crashing if malloc returns a NULL is your fault, did musl really feel the need to accommodate bad implementations like this or is this just AI speculation?
"unfortunately, a large portion of programs which call crypt are not prepared for its failure and do not check that the return value is non-null before using it. thus, always "succeeding" but giving an unmatchable hash is reportedly a better behavior than failing on error." [1]
[1] https://git.musl-libc.org/cgit/musl/commit/?id=b3c4cc121f70f...
Such a questionable decision! "Rather than let a bug in a caller (not handling the valid null output and thus crashing) surface, let's just silence it and pretend that the caller is correct! A potential clash in some situations (resulting in this vulnerability) is better than a crash!"
Maybe I should update my plea to fail loudly (https://alejo.ch/3gk) to list this as an example.
2 replies →
> did musl really feel the need to accommodate bad implementations like this
I don't know about this specific case but in general? Yes, totally. No one is going to fix all existing software that depends on libs behaving exactly as glibc, you have to accept some trade-offs. Either your new implementation stops being perfectly simple and clean, or no one uses it.
it's not "glibc behavior", returning null on error is in the spec
1 reply →
See also: Hyrum's Law
Thanks you to the team that found this. Exceptionally well explained and funny at the same time. What is most important is how people just checking stuff( remember the .xz hack that was tried?) has save the internets ass so many times. We should give good security researchers UBI for life for just doing good work.
They list OpenWRT as affected -- but by default OpenWRT runs dropbear not OpenSSH.
Dropbear is unaffected.
It would be nice if the passwd crypt functions didn't have timing dependence on password length. Then there'd never need to be hacky workarounds like what musl did, and various tricks to avoid exposing whether users exist on systems.
No idea if it's likely to happen though.
It's time to switch to OpenBSD
So, nobody fuzz tested sshd before?!?!
> So, nobody fuzz tested sshd before?!?!
OpenBSD probably did (which doesn't use musl), RedHat probably did (or some other distro using glibc), but they all don't use musl.
Yeah, I am mystified as to how this could have happened.
It's code written by meat shrugs
We use Alpine quite a bit so I built our team ssh-audit[1]. Feel free to use it if you want.
1. https://news.ycombinator.com/item?id=50383006
Man, that author is really committed to that lowercase aesthetic. Even the entire personal website. I wonder what's going on with that? We used to do that back on AIM informally. I mostly just see Gen Z and young kids still doing that now. Pretty crazy how all the stuff that used to be trendy comes back.
I don’t think you should let it bother you. I still found it more pleasant to read than the average article’s LLM slop
minor nit: for Alpine's root, the * came with 3.16 in 2022, so that path is 4 years old.
the sshd/musl part dates to the 2016 CVE-2016-6210 fix, so other * accounts (key-only users, LDAP users) have been open for 10 years.
the website is amazing, I just think these details would clarify things.
I learned maybe a year in software, should not match/happen are famous last words.
This is just negligent. Not a fan of rewrite in rust, but I find it highly unlikely this would happen in that community.
Rust tells you off if you don’t handle Results. Sure, you can still just unwrap() it and have it panic, but at least that’s a specific decision instead of forgetting to check for NULL like in this case.
My antivirus won't let me access the site, stating that it contains a threat: URL: Phishing.
Managed to reproduce this, with a lab[0] you can play with.
[0] https://github.com/leonjza/sshhole-lab
Repro in a Docker container: https://github.com/predkambrij/sshhole-repro
Not the worst site, but it's clearly a publicity machine from wakefield.co, whom I haven't heard of. Also, exploitation conditions are super narrow, albeit, any password is wild.
Reminds one of https://www.openssh.org/txt/buffer.adv.
Horrors behind comprehension all around. The code, the error case being “*”, the loose patches plastered over. musl and C are unserious
I just created an Alpine container, did "apk add openssh" and this doesn't work for me. I assume it's been updated already.
Related discussion: [0]
[0] https://news.ycombinator.com/item?id=50377221
Can we just have a site with the text and images without the awkward scrolling crap? Hijacking basic features is an instant click away from sites for me
Edit: It seems to work fine on mobile. I'm not sure why the scrolling behaviour was added on the desktop version of the site
This site is beautiful I'm going to say
how come such a thing can exist for more than a decade and get unnoticed? Weird times.
Can we all marvel at the site. The author says they did the discovery on November 3. Since then they apparently bought a domain, set up the webpage, created animations, and a cartoon. All kinds of stuff, and got it to explode on hackernews. A span of 2-3 days. How long ago was this not even conceivable?
That's kind of shocking. I guess the swarm of AI related vulnerability findings continues.
It's shocking that it took so long with the amount of investment and hype you'd expect this kind of bugs to be already fixed, but it still need a human to tell the agents what to do, which could also work with any fuzzing framework
I really like creativity in the name, the way website built and the content
"tl;dr" was a bit of a downgrade though, but that's fine
it's a nice way to promote the company behind it
Am I missing something or is this written up to seem a lot worse than it actually is?
It starts with a bold claim: "ssh accounts that should not accept a password will accept any password between 256 and 1024 characters". But then it keeps adding qualifiers:
- sshd compiled with musl
- not using PAM
- password logins enabled
- at least one account with '*' as the password "hash"
- at least one OTHER account with a valid password
- server is aligned with 2 ley lines
- hacker is wearing a blue t-shirt and green socks
Ok, the last two are an exaggeration, but still. The conditions are all ANDed, miss a single one and you're fine.
On top of that, as someone else in these comments reminded, the target account needs to have a valid login shell or other exploitable program assigned to it, and these passwordless accounts tend to have nologin or something similar.
To me this seems to fall on the "technically possible, practically very unlikely" bucket, kinda like the PHP vulnerability from the other day. Maybe this has gone unnoticed for so long because it just doesn't happen.
There are a lot of systems that meet those requirements.
> - sshd compiled with musl
So alpine as well as a bunch of other embedded Linuxes. Huge install base.
> - not using PAM
What you get if you try to have a minimal setup. Default sshd config if you install from source.
> - password logins enabled
The default. Super common.
> - at least one account with '*' as the password "hash"
Almost every Linux system.
> - at least one OTHER account with a valid password
Almost every Linux system.
So sure, there are caveats, but they cover a massive install base.
This is a super bad bug
How many alpine installs are running sshd? Most containers using it as a base image aren't. Embedded Linux is unlikely to be running it, too. And if one IS running sshd, how often is it with unchanged defaults? I don't buy the inference here.
I'll concede that embedded linux might be my blindspot here. Maybe cheap web hosting, too, but that seems much less likely to be using musl.
16 replies →
For what it's worth, default Alpine installs don't have an account with * as its password hash if you use the root account as your main one. Not sure if that's also true for ones where you set up a user, maybe those ones are vulnerable.
I'm genuinely curious what's the Venn diagram here because I feel a lot of embedded systems precisely have one condition but not the other, e.g. password yes but not necessarily multiple accounts, e.g root only, while using musl. Namely almost all for almost all, but if only one is not overlapping another then it's actually very few.
>> - at least one account with '*' as the password "hash"
> Almost every Linux system.
But not, notably, on Alpine, which uses '!' instead.
Edit: or maybe yes on Alpine? I checked my desktop and it has zero *s in shadow, but Alpine's post seems to say they do use * sometimes
1 reply →
does this really matter? It could be used to do something devastating. The number of systems that match those requirements is only one factor. If just a handful of those are highly critical for humanity it could lead to disaster. Eg. hacks into fridges, centrifuges etc.
Updates since this was posted:
- blue t-shirts and green socks have been added to the default loadout for new hackers, since it's optimal according to AI and costs nothing to the attacker in terms of compute
- there are so many ley line "maps" out there that basically any server placement counts as aligned with at least two, so this one's free also
5 replies →
> password logins enabled
Disabling sshd password logins is security 101, and the default for some cloud provided images these days. If password logins haven't been disabled, what other doors have been left open anyway?
Now the question for the man: would fil-c prevent it :D
> > - password logins enabled
> The default. Super common.
This always seems bonkers to me. I think SSHD should give you ONE password free connection of setup to copy your key over and then it just shuts the door behind it.
2 replies →
Alpine Linux (which uses musl) images are popularly deployed by those who use docker, and apparently the issue is there on the default install.
Not sure whether the PHP vulnerability actually had been exploited, but I'm pretty sure there are literally thousands of machines out there affected by this ssh issue right now as we speak.
This isn't going to end well.
---
Addendum (which may or may not be related to the above) --
Everyone has heard of the tale of the boy who cried wolf. The main moral of the story is not to troll your fellow townsfolks so they don't distrust you. But everyone with a sane mind gets that.
The (IMHO) under-appreciated lesson is actually: don't be those villagers. Believe stories you are given, even if you might look like a fool for believing in them. Don't assume what happens is the same as what happened in the past. Look at the facts yourself. Trust and verify.
You don't typically run sshd in containers, instead you only run whatever single service you set up your container for.
Sure there are native Alpine installs too, but that is far less common of a use case.
Who's out there running sshd inside a docker container? It's almost MORE work than exec-ing into the container.
8 replies →
Alpine in containers is usually not affected. Most images/containers haven't ssh/sshd installed at all. Remote access is usually done from the host system.
You could start with trust and verifying what you wrote first, the popular usage of alpine in docker containers don't run sshd.
1 reply →
Really, "password auth enabled" should be enough to be a non starter these days.
you'd be surprised
only 1) is a narrowing filter. the rest is basically the majority.
Most linux distros use PAM, so 2 is also a narrowing filter. Debian, Ubuntu, Fedora, Arch, etc all use PAM.
5 replies →
Yes - immediately had me in a mild panic that I’d missed another AI discovered crit bug!
Had to quickly scan the rest of the homepage of HN to confirm.
I suppose there is the chance you get to prep a system setup at one time, so you can access it at a later time?
Yes, 100% written to oversell it as hard as possible. Still a vuln but karma farming too.
Yes, missing the rest of the world that does not conform to your self-references.
> - hacker is wearing a blue t-shirt and green socks
You had me there :D
I think you're missing the reason why it's relevant now.
LLMs will find it.
They work endlessly at finding the 'and' scenarios.
This should have been closed up a long time ago.
Every time I see one of these named exploit sites I always think, "Hey look at this guy, he wants attention."
> kinda like the PHP vulnerability from the other day
What? Pretty sure Cloudflare detected widespread attacks almost immediately after it was published. We are probably lucky that not much more happened because of responsible disclosure and the large media attention led to widespread admin awareness.
It’s irrelevant how widespread it is. It’s a bad bug that should be fixed. Why do people always say stuff like this? If you’re running a system that is exploitable then you deserve to know about it and to be provided a fix. It doesn’t matter all that much how widespread it is. The CVE system doesn’t exist to cater only to highly exposed systems.
It matters how widespread it is because humans have limited time and resources to solve problems. If these conditions only exist in 0.001% of installs, is it worth the man hours to fix it? Alpine may think not, because like others have mentioned, there are likely other higher probability vulnerabilities being exposed in order to meet the conditions for this to even matter. A judgement call can (and clearly has?) be made to prioritize other work.
The good news is that considering Alpine is open-source, you can be the one to make the judgement call that it is worth fixing and go out there and fix it ;)
4 replies →
Openwrt is mentioned to be affected and we use openwrt in our product. Looks like our configuration is different enough that we are unaffected, but still I thought I mention the vuln in our company chat. Turns out that M$ Teams does not let me post the link. Like in China, the party knows what news are good for you.
(Of course I could post it a bit obfuscated)
I’m a smart person and none of this makes any sense to me. It’s like purposefully written poorly.
Agreed. Much better explained here: https://news.ycombinator.com/item?id=50379342
I'm starting to realize that trusting some random people who wrote the gruddiest, boringest, most torturous low-level stuff might have been a bad plan.
At this point I feel embarrassed in front of the AIs.
This is a super duper annoying site for people who just want to know if their systems are vulnerable, after patching like a dozen critical vulnerabilities in other software over the last week and are almost at the breaking point of resigning.
Can we have a “don’t treat me like five/toddler” / “idgaf about the explanation” quick link to a section for adults who need to patch systems?
(And if your reaction is “just scroll down, stupid”, I’m happy to report that with the scroll jacking, I scroll not enough / past the critical sections nine out of ten times.)
Edit: You may want to read https://sshhole.com/#exposure.
Thanks for mentioning that before I tried to scroll down. Forewarned, I was able to click on the "2026" link once the left-handed "scrollbar" (not the right word but I can't figure out what the right word should be) appeared, which let me jump past most of the verbiage and get to the "are you affected?" section just a couple pages later.
[flagged]
What this tells me is that musl+openssh-server is an incredibly uncommon setup, as "at least 257 character string" is literally the third thing I try after "admin" and "12345" when greeted with a password input.
If The UNIX-HATERS Handbook should ever get updated, this deservers its own chapter.
At this point we should all agree that software engineering is an elaborate well paid joke.
Joke's on you, my busybox only supports generating MD5 hashes so I'm safe.
Theory: “With enough eyes, all bugs are shallow.”
Corollary: “Nobody looks”
The proof of this corollary is left as an exercise for the reader.
The best part of this exploit is undoubtedly the name, *sshole.
This AI slop write-up is so fucking annoying to read. I don't even mind the AI generated website. Why can't you even bother to spend like an hour to properly sum up the exploit yourself?
The real name of this is sshole - the is the "star" of the show, after all.
*sshole, the * is the star of the show
You can use \ to escape formatting.
Ugh! HN got me, too
I would like to read this but the scrolling is completely broken on iPad Safari
I liked that this site explained the vulnerability in an easy-to-understand way.
I can't understand it. All this scrolling and pictures are distracting me, can't concentrate really.
I think it varies from person to person. Since I'm kind of from the TikTok generation, I don't have much trouble reading this kind of animation, but some people prefer static sites. That's just how it is.
I also found scroll-triggered animations hard to read at first, but these days a lot of sites use them, so I've gotten used to them.
1 reply →
Yep, scrolling through my phone and reading slowly to understand. I'm seeing distracting animations below what I'm reading. And when I go to them, there is no way to restart them. So it completely misses the point.
Am I the only one who is shocked that such important and security-relevant software just does unorthodox stuff like "well let's just return `*` if it fails" and other kind of "hacks" ?! The developers must know that this is very bad software engineering and will definitely cause issues at _some_ point in time, no?!
almost seems like a planted backdoor in alpine.
expect more of these
This is bottom-up branding I can get behind. ()*()
Good thing my yocto images use glibc...
came for the name, stayed for the story
Shouldn't the title of this post be *sshole? An asterisk isn't disallowed in post titles, is it?
He mentions that circa 2019, Debian's libcrypt stopped returning NULL and began returning the 2-character string '*0' - he doesn't elaborate, but this was part of a larger migration by many distros. glibc deprecated its own libcrypt so a more agile project that 'could move faster' would take its place [0]. This migration happened in Fedora 28 [1], Arch [2], and Debian bullseye [3].
Unless you specifically compile it with --disable-failure-tokens, on an error libxcrypt will return that 2-character failure token string. [4] [5] This behavior is very old [6], and hasn't been POSIX-compliant since at least POSIX.1-2001, which is based on SUSv2, which is from 1997.
So they've knowingly kept this non-standard backwards compatibility cludge for almost 30 years, and continue to claim it "isn't well standardized" in their man page despite POSIX and glibc existing, but at least it didn't bite them in the asterisk like it did musl. (Yet.) There's a new issue about removing this behavior now. [7]
[0] https://sourceware.org/legacy-ml/libc-alpha/2017-08/msg01257...
[1] https://fedoraproject.org/wiki/Changes/Replace_glibc_libcryp...
[2] https://bugs.archlinux.org/task/67312
[3] https://bugs.debian.org/941853
[4] https://manpages.debian.org/stretch/manpages-dev/crypt.3.en....
[5] https://manpages.debian.org/bullseye/libcrypt-dev/crypt.3.en...
[6] https://github.com/besser82/libxcrypt/commit/a1a3cba626843f3...
[7] https://github.com/besser82/libxcrypt/issues/226
Was this exploited in the wild?
Haha insane, great find and write up
something interesting to note here is that while this isn't a memory safety vulnerability, a modern systems programming language with better error handling could have easily avoided this whole situation. error handling in C is a PITA, and we're paying the cost of decades of pretending that it's not a problem
What did you just call me?
I need to check my fridge...
Wow seriously no unit or integration test for this?
All that site is lacking is marquee and furry creatures. AI slop sites are so horrible with their blinking animations, l33t dark background and chaotic text.
rich felker was such a f* (funny) guy back in the mplayer days too
never changed i guess
Love the website
One h or two?
SSH olé
One more reason to build your (own) Docker images on 'FROM scratch' (basically nothing).
No, the Alpine containers don't come with sshd preinstalled. You'd have to manually install via dockerfile and leave it unconfigured.
No one runs sshd in containers, at least I hope so. What would even be the point when you have docker/kubectl commands available?
Using LXC (not the same as docker) and running openssh inside the container.
This is fine :)
What is this idiotic manner of typing full articles in lowercase? Don't people learn basic orthography in school anymore? This is fucking unreadable.
I refuse to believe this is not a backdoor.
I desperately want someone to create a single-letter subdomain here.
at this point my only reaction is "are you fucking kidding me"
It’s dumb and negligent to be missing “PasswordAuthentication no” in 2026. It’s been so for at least ten years. OpenSSH should have turned off password auth in the defaults years ago.
just thank god for dropbear
This one feels negligent. How does this go uncaught for 10 years? How did it make it into production period?
We need to re-think EVERY assumption about security.
quick fix?
> how did I ran into it? [...] i had agent swarms. a lot of them. let's not get into it
Can we please now normalize AI usage in development? Just some other day people asked for "NO AI used" badges for software as measure for quality, and praised i.e. NetBSD for being AI-free. Yes, we come from a phase of AI slop, but it has become obvious that there are plenty of legit usecases where AI beats human-only effort nowadays.
You've got a great point. But let's also acknowledge that there's a pretty stark contrast between these:
- I found a way to get AI to help me while I made this
- I asked AI to make this
The people you're objecting to are complaining about the latter, this is an example of the former.
There's a decent-size "any AI usage is theft" contingent that reacts dramatically to any usage of AI whatsoever.
Not sure why but a lot of people seem to be missing that you don't have to look at AI as a binary "you're either with us or you're with the terrorists."
Just do HITL (human in the loop). Use an LLM to help you author, but steer the AI in whatever direction you want. It's like having a slightly delusional copy of yourself to pair program with. If you take the time to provide an LLM with some additional skills files and the like, with you in the pilot seat, you can get solid results (it doesn't look as cool as a hands free agent setup but the quality far surpasses anything they can do solo).
It's a stupid and wasteful way to automate the process.
> Can we please now normalize AI usage in development?
1. It is already normalized outside of a tiny, but very vocal bubble that lives mostly on Bluesky (sometimes also on Reddit/HN/Mastodon).
2. I don't think that even the most staunch AI hater would object to using AI for security research as long as all the artifacts (code, docs) that were produced based on the research are human-written. "Yes, there is a bug in my program, but I refuse to fix it, because I know that you found it using AI" would be a pretty weird position. Not sure, how would you defend it.
I feel like there is a lot of pushback from delusional engineers, that won't accept their coding skills have been rendered useless - so they come up with tons of reasons to avoid AI, because it just has to be bad. Anything to hold up the ilusion you are an well-paid white-collar worker that can pick employers anytime. And I say this myself coding since the late 90s and making a living off it - it is a painful pill to swallow, but the genie does not go back in the box so just accept it and move on.
This is a pretty well done AI generated website, congrats you have proven it's not solely for slop.
Are articles like this deliberately designed to make it to the frontpage of HN?
what was done to report & patch the vuln? Why publish it in this format? It’s presented like a whistleblower who’s exhausted every approach, but the effort to do the right thing is not clear.
does this have CVE or something?
cve-2026-123158
> on alpine's official aws and azure images that's the default alpine user, and it has passwordless doas. so it's root
good fucking god. if I get this correctly... you can insta-own any aws/azure instance that was set up with alpine as long as it has ssh enabled, which it probably has.
Tbh the most disappointing part of this is musl (and later glibc) changing the function to stop callers crashing because they didn't RTFM.
jesus christ wtf
Can we hear something from those who say that using port-knocking is "security through obscurity"?
Really: from the XZ backdoor in SSH (which went live on some distros) to this...
What's you rationale for calling port-knocking "security through obscurity" and not "defense in depth"?
Put it this way: if I advertize that my server is using port-knocking, that the sequence changes every 30 seconds (like TOTP) and that as soon one sequence is used, it's unusable (so you've only got one attempt to SSH in every 30 seconds, even if you know the sequence)...
Where's the security through obscurity?
But really: even if had musl and PasswordAuthentication allowed (I don't have neither), how would an attacker gets through the port knocking?
More fundamentally the question is: in which way is your system that does not use port knocking more secure than the one who do?
(remember: we're talking about remote exploits by simply having exposed TCP ports open or SSH listening)
Wanna have a talk too about the "solutions" proposed by "security experts" here "because SSH is too old" and that happen to be crufts on top of SSH that had remote root exploitable backdoors? (sorry, holes)
[flagged]
Disclosure: I work at Detectify. A few practical notes for anyone checking their own machines:
- It affects Alpine's openssh-server (built without PAM), or openssh-server-pam with UsePAM no, when PasswordAuthentication is enabled. Alpine says every release they checked is affected, 3.16 through 3.24 and edge.
- Fixed openssh packages are out for all of those, including the older 3.16-3.20 releases. Upgrade and restart sshd (rc-service sshd restart). If you can't upgrade yet, set PasswordAuthentication no.
- To check a box: apk info -v openssh-server for the version, and sshd -T | grep -iE 'usepam|passwordauthentication' for the effective config.
- Containers built FROM alpine that run sshd are easy to miss in an inventory. So are the official tiny-cloud images (3.18.11 through 3.24.3), which ship an alpine user with no password and passwordless doas, so a bypass there means root.
- After patching, look in /var/log/messages for sshd lines saying "Accepted password for" an account that has no password.
[flagged]
[dead]
[flagged]
[dead]
[flagged]
[flagged]
[dead]
[flagged]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
what’s the moral of the story?
alpine stocks are crashing?
musl stocks are crashing?
glibc stonks are up
alternative implementations in general slightly down
[flagged]
> Also, what's with this damn trend of typing in lowercase only?
Anticapitalism.
You made my morning. Got a new dad joke.
you could make a strong case for that!!!
Anticapitalism that uses an army of clankers?
1 reply →
[flagged]
wtf did I just read?