Comment by Meneth
2 months ago
This happens because there's no auditing of new packages or versions. The distro's maintainer and the developer is the same person.
The general solution is to do what Debian does.
Keep a stable distro where new packages aren't added and versions change rarely (security updates and bugfixes only, no new functionality). This is what most people use.
Keep a testing/unstable distro where new packages and new versions can be added, but even then added only by the distro maintainer, NOT by the package developers. This is where the audits happen.
NPM, Python, Rust, Go, Ruby all suffer from this problem, because they have centralized and open package repositories.
This is a culture issue with developers who find it OK to have hundreds of (transitive) dependencies, and then follow processes that, for all intents and purposes, blindly auto update them, thereby giving hundreds of third-parties access to their build (or worse) execution environments.
Adding friction to the sharing of code doesn't absolve developers from their decision to blindly trust a ridiculous amount of third-parties.
I find that the issue is much more often not updating dependencies often enough with known security holes, than updating too often and getting hit with a supply-chain malware attack.
There have been several recent supply chain attacks that show attackers are taking advantage of this (previously sensible) mentality. So it is time to pivot and come up with better solutions before it spirals out of control.
1 reply →
Not updating is the other side of the same problem: library owners feel it is ok to make frequent backwards-compatibility breaking changes, often ignoring semver conventions. So consumers of their libraries are left with the choice to pin old insecure versions or spend time rewriting their code (and often transitive dependency code too) to keep up.
This is what happens when nobody pays for anything and nobody feels they have a duty to do good work for free.
5 replies →
It's not unreasonable to trust large numbers of trustworthy dependency authors. What we lack are the institutions to establish trust reliably.
If packages had to be cryptographically signed by multiple verified authors from a per-organization whitelist in order to enter distribution, that would cut down on the SPOF issue where compromising a single dev is enough to publish multiple malware-infested packages.
"Find large numbers of trustworthy dependency authors in your neighborhood!"
"Large numbers of trustworthy dependency authors in your town can't wait to show you their hottest code paths! Click here for educational livecoding sessions!"
1 reply →
It IS unreasonable to trust individual humans across the globe in 100+ different jurisdictions pushing code that gets bundled into my application.
How can you guarantee a long trusted developer doesn't have a gun pointed to their head by their authoritarian govt?
In our B2B shop we recently implemented a process where developers cannot add packages from third party sources - only first party like meta, google, spring, etc are allowed. All other boilerplate must be written by developers, and on the rare occasion that a third party dependency is needed it's copied in source form, audited and re-hosted on our internal infrastructure with an internal name.
To justify it to business folks, we presented a simple math where I added the man-hours required to plug the vulnerabilities with the recurring cost of devsecops consultants and found that it's cheaper to reduce development velocity by 20-25%.
Also devsecops should never be offshored due to the scenario I presented in my second statement.
2 replies →
Problem is that beyond some threshold number of authors, the probability they're all trustworthy falls to zero.
1 reply →
> This is a culture issue with developers who find it OK to have hundreds of (transitive) dependencies, and then follow processes that, for all intents and purposes, blindly auto update them
I do not know about NPM. But in Rust this is common practice.
Very hard to avoid. The core of Rust is very thin, to get anything done typically involves dozens of crates, all pulled in at compile time from any old developer implicitly trusted.
The same is true for go and for java.
1 reply →
See auto update bots on Github. https://docs.github.com/en/code-security/dependabot/dependab... And since Github does it, it must be a good thing, right? Right???
Unfortunately that's almost the whole industry. Every software project I've seen has an uncountable amount of dependencies. No matter if npm, cargo, go packages, whatever you name.
Every place I ever worked at made sure to curate the dependencies for their main projects. Heck, in some cases that was even necessary for certifications. Web dev might be a wild west, but as soon as your software is installed on prem by hundreds or thousands of paying customers the stakes change.
1 reply →
Zero-external-dependency Go apps are far more feasible than Rust or Node, simply because of the size and quality of the standard library.
15 replies →
>absolve developers
Doesn't this ultimately go all the way up to the top?
You have 2 devs: one who mostly writes their own code, only uses packages that are audited etc; the other uses packages willy nilly. Who do you think will be hired? Who do you think will be able to match the pace of development that management and executives demand?
Rather than adding friction there is something else that could benefit from having as little friction as sharing code: publishing audits/reviews.
Be that as it may, a system that can fail catastrophically will. Security shouldn't be left to choice.
There is another related growing problem in my recent observation. As a Debian Developer, when I try to audit upstream changes before pulling them in to Debian, I find a huge amount of noise from tooling, mostly pointless. This makes it very difficult to validate the actual changes being made.
For example, an upstream bumps a version of a lint tool and/or changes style across the board. Often these are labelled "chore". While I agree it's nice to have consistent style, in some projects it seems to be the majority of the changes between releases. Due to the difficulty in auditing this, I consider this part of the software supply chain problem and something to be discouraged. Unless there's actually reason to change code (eg. some genuine refactoring a human thinks is actually needed, a bug fix or new feature, a tool exposed a real bug, or at least some identifiable issue that might turn into a bug), it should be left alone.
I agree with this and it's something I've encountered when just trying to understand a codebase or track down a bug. There's a bit of the tail wagging the dog as an increasing proportion of commits are "meta-code" that is just tweaking config, formatting, etc. to align with external tools (like linters).
> Unless there's actually reason to change code (eg. some genuine refactoring a human thinks is actually needed, a bug fix or new feature, a tool exposed a real bug, or at least some identifiable issue that might turn into a bug), it should be left alone.
The corollary to this is "Unless there's actually a need for new features that a new version provides, your existing dependency should be left alone". In other words things should not be automatically updated. This is unfortunately the crazy path we've gone down, where when Package X decides to upgrade, everyone believes that "the right thing to do" is for all its dependencies to also update to use that and so on down the line. As this snowballs it becomes difficult for any individual projects to hold the line and try to maintain a slow-moving, stable version of anything.
I'm using difftastic, it cuts down a whole lot of the noise
https://difftastic.wilfred.me.uk/
This looks good! Unfortunately it looks like it also suffers from exactly the same software supply chain problem that we need to avoid in the first place: https://github.com/Wilfred/difftastic/blob/master/Cargo.lock
Edit: also, consider how much of https://github.com/Wilfred/difftastic/commits/master/ is just noise in itself. 15k commits for a project that appears to only be about four years old.
3 replies →
In Rust we have cargo vet, where we share these audits and use them in an automated fashion. Companies like Google and Mozilla contribute their audits.
I wish cargo went with crev instead, that has a much better model for distributed code audits.
https://github.com/crev-dev/
It's too bad MS doesn't own npm, and/or GitHub repositories. Wait
Nuget, Powershell gallery, the marketplaces for VSCode/VS/AZDo and the Microsoft Store too. Probably another twenty.
They collect package managers like funko pops.
I'm not quite sure about the goal. Maybe some more C# dev kit style rug-pulls where the ecosystem is nominally open-source but MS own the development and distribution so nobody would bother to compete.
1 reply →
And it's a great idea, similar thematically to certificate transparency
How to backport security fixes to vetted packages?
I'd like to think there are ways to do this and keep things decentralized.
Things like: Once a package has more than [threshold] daily downloads for an extended period of time, it requires 2FA re-auth/step-up on two separate human-controlled accounts to approve any further code updates.
Or something like: for these popular packages, only a select list of automated build systems with reproducible builds can push directly to NPM, which would mean that any malware injector would need to first compromise the source code repository. Which, to be fair, wouldn't necessarily have stopped this worm from propagating entirely, but would have slowed its progress considerably.
This isn't a "sacrifice all of NPM's DX and decentralization" question. This is "a marginally more manual DX only when you're at a scale where you should be release-managing anyways."
> two separate human-controlled accounts to approve any further code updates.
Except most projects have 1 developer… Plus, if I develop some project for free I don't want to be wasting time and work for free for large rich companies. They can pay up for code reviews and similar things instead of adding burden to developers!
I think that we should impose webauthn 2fa on all npm accounts as the only acceptable auth method if you have e.g., more than 1 million total downloads.
Someone could pony up the cash to send out a few thousand yubikeys for this and we'd all be a lot safer.
Why even put a package download count on it? Just require it for everything submitted to NPM. It's not hard.
8 replies →
Even the simplest "any maintainer can click a push notification on their phone to verify that they want to push an update to $package" would have stopped this worm in its tracks!
How would that work for CI release flows? I have my Rust crates, for example, set up to auto-publish whenever I push a tag to its repo.
PyPI already has this. It was a little bit annoying when they imposed stricter security on maintainers, but I can see the need.
Pypi did that, i got 2 google keys for free. But I used them literally once, to create a token that never expires and that is what I actually use to upload on pypi.
(I did a talk at minidebconf last year in toulouse about this).
If implemented like this, it's completely useless, since there is actually no 2fa at all.
Anyway the idea of making libre software developers work more is a bad idea. We do it for fun. If we have to do corporate stuff we want a corporate salary to go with.
You can use debian's version of your npm packages if you'd like. The issues you're likely to run into are: some libraries won't be packaged period by debian; those that are might be on unacceptably old versions. You can work around these issues by vendoring dependencies that aren't in your distro's repo, ie copying a particular version into your own source control, manually keeping up with security updates. This is, to my knowledge, what large tech companies do. Other companies that don't are either taking a known risk with regards to vulnerabilities, or are ignorant. Ignorance is very common in this industry.
Distros are struggling with the amount of packages they have to maintain and update regularly. That's one of the main reasons why languages built their own ecosystems in the first place. It became popular with CPAN and Maven and took off with Ruby gems.
Linux distros can't even provide all the apps users want, that's why freshmeat existed and we have linuxbrew, flatpak, Ubuntu multiverse, PPA, third party Debian repositories, the openSUSE Buildservice, the AUR, ...
There is no community that has the capacity to audit and support multiple branches of libraries.
The lack of an easy method to automatically pull in and manage dependencies in C/C++ is starting to look a lot more like a feature than a bug now.
Author of Odin is also against adding a package manager: https://www.gingerbill.org/article/2025/09/08/package-manage...
But there's so much UB in C++ that can be exploited that I doubt attackers lament the lack of a module system to target. ;)
To be clear, Debian does not audit code like you might be suggesting they do. There are checks for licensing, source code being missing, build reproducibility, tests and other things. There is some static analysis with lintian, but not systematically at the source code level with tools like cppcheck or rust-analyzer or similar. Auditing the entirety of the code for security issues just isn't feasible for package maintainers. Malware might be noticed while looking for other issues, that isn't guaranteed though, the XZ backdoor wasn't picked up by Debian.
https://lintian.debian.org/
> The general solution is to do what Debian does.
The problem with this approach is that frameworks tend to "expire" pretty quickly and you can't run anything for too long on Debian until the framework is obsolete. What I mean by obsolete is Debian 13 ships with Golang 1.24, A year from now it's gonna be Golang 1.26 - that is not being made available in trixie. So you have to find an alternative source for the latest golang deb. Same with PHP, Python etc. If you run them for 3 years with no updated just some security fixes here and there, you're gonna wake up in a world of hurt when the next stable release comes out and you have to do en-masse updates that will most likely require huge refactoring because syntax, library changes and so on.
And Javascript is a problem all by itself where versions come up every few months and packages are updated weekly or monthly. You can't run any "modern" app with old packages unless you accept all the bugs or you put in the work and fix them.
I am super interested in a solution for this that provides some security for packages pushed to NPM (the most problematic repository). And for distributions to have a healthy updated ecosystem of packages so you don't get stuck who knows for how long on an old version of some package.
And back to Debian, trixie ships with nginx 1.26.3-3+deb13u1. Why can't they continuously ship the latest stable version if they don't want to use the mainline one?
> Keep a stable distro where new packages aren't added and versions change rarely (security updates and bugfixes only, no new functionality). This is what most people use.
Unfortunately most people don't want old software that doesn't support newer hardware so most people don't end up using Debian stable.
It'd be interesting to see how much of the world runs on Debian containers, where most of the whole "it doesn't support my insert consumer hardware here" argument is completely moot.
I don't know why you went with hardware.
Most people don't want old software because they don't want old software.
They want latest features, fixes and performance improvements.
What hardware isn't supported by Debian stable that is supported by unstable?
Or is this just a "don't use Linux" gripe?
I haven't had much problems prior but Blackwell support was really buggy for the first two weeks.
Enable the Backport sources. The recent kernels there have supported all my modern personal devices.
> Unfortunately most people don't want old software
"old" is a strange way to spell "new, unstable, and wormed".
I want old software. Very little new features are added to most things i care about, mostly it is just bloat, AI slop, and monthly subscription shakedowns being added to software today.
So, who is going to audit the thousands of new packages/versions that are published to npm every day? It only works for Debian because they hand-pick popular software.
Maybe NPM should hand pick popular packages and we should get away from this idea of every platform should always let everyone publish. Curation is expensive, but it may be worthwhile for mature platforms.
This is maybe where we could start getting into money into the opensource ecosystems.
One idea I've had is that publishing is open as today, but security firms could offer audit signatures.
So a company might pay security firms and only accept updates to packages that have been audited by by 1,2,3 or more of their paid services.
Thus money would be paid in the open to have eyes on changes for popular packages and avoid the problem of that weird lone maintainer in northern Finland being attacked by the Chinese state.
Errr, you! If you brought the dependency, it is now your job to maintain it and diff every update for backdoor.
I've been arguing a couple of times that the 2 main reasons people want package management in languages are
1. Using an operating system with no package management 2. Poor developer discipline, i.e. developers always trying to use the latest version of a package.
So now we have lots of poorly implemented language package managers, docker containers on top being used as another package management layer (even though that's not their primary purpose but many people use the like that) and the security implications of pulling in lots of random dependencies without any audit.
Developing towards a stable base like Debian would not be a pancea, but alliviate the problems by at least placing another audit layer in between.
Nope. It's because:
1. You don't want to tie your software to the OS. Most people want their software to be cross-platform. Much better to have a language-specific package manager because I'm using the same language on every OS. And when I say "OS" here, I really mean OS or Linux distro, because Linux doesn't have one package manager.
2. OS package managers (where they even exist), have too high a bar of entry. Not only do you have to make a load of different packages for different OSes and distros, but you have to convince all of them to accept them. Waaay too much work for all but the largest projects.
You're probably going to say "Good! It would solve this problem!", but I don't think the solution to package security is to just make it so annoying nobody bothers. We can do better than that.
I actually agree in the context of user software people often want the latest and that Windows and OS don't have proper package management is an issue.
However we are talking in the context of NPM packages which by the vast majority would be running inside a container on some server. So how could that software not use a stable Debian base for example.
And arguing that package management is to complicated is a bit ridiculous considering how many workloads are running in docker containers which I'd argue are significantly more complex
It doesn't matter if the operating system I personally use has a good package manager, I need to release it in a form that all the people using it can work with. There are a lot of OSes out there, with many package managers.
Even if we make every project create packages in every package manager, it still wouldn't add any auditing.
Exactly, in a way Debian (or any other distro) is an extended standard library.
Yeah, after seeing all of the crazy stuff that has been occurring around supply chain attacks, and realizing that latest Debian stable (despite the memes) already has a lot of decent relatively up-to-date packages for Python, it's often easier to default to just building against what Debian provides.
Right. Like NPM, Debian also supports post-install hooks for its packages. Not great (ask Michael Stapelberg)! But this is still a bit better than the NPM situation because at least the people writing the hooks aren't the people writing the applications, and there's some standards for what is considered sane to do with such hooks, and some communal auditing of those hooks' behavior.
Linux distros could still stand to improve here in a bunch of ways, and it seems that a well-designed package ecosystem truly doesn't need such hooks at the level of the package manager at all. But this kind of auditing is one of the useful functions of downstream software distros for sure.
> NPM, Python, Rust, Go, Ruby all suffer from this problem, because they have centralized and open package repositories
Can you point me to Go's centralized package repository?
https://pkg.go.dev/
Which works together with
https://proxy.golang.org/
That's a doc site and a pull-through cache, neither is a package repository
https://github.com/
git isn't centralized nor a package repository
For what it's worth, our code is on GitLab
12 replies →
Pretty unfeasible with the variety of packages/ecosystems that get created. You'd either ending up requiring a LOT of dev time looking over packages on the maintainer end, or basically having no packages people need to use in your repository.
Finding the balance of that seems to me like it'd be incredibly difficult.
In practice, my experience is that this ends up with only old versions of things in the stable package repos. So many times I run into a bug, and then find out that the bug has been fixed in a newer version but it isn't updated in the stable repo. So now you end up pulling an update out of band, and you are in the same boat as before.
I don't know how you avoid this problem
You're overestimating the amount of auditing these distros do for the average package; in reality there is very little.
The reason these compromised packages typically don't make it in to e.g. Debian is because this all tends to be discovered quite quickly, before the package maintainer has a chance to update it.
security updates and bugfixes only
Just wondering: while this is less of an attack surface, it's still a surface?
NX NPM attack (at least the previous wave which targetted tinycolor) relied on running post-install scripts. Go tooling does not give you ways to run post-install scripts, which is much more reasonable approach.
> The general solution is to do what Debian does.
If you ask these people, distributions are terrible and need to die.
Python even removed PGP signatures from Pypi because now attestation happens by microsoft signing your build on the github CI and uploading it directly to pypi with a never expiring token. And that's secure, as opposed to the developer uploading locally from their machine.
In theory it's secure because you see what's going in there on git, but in practice github actions are completely insecure so malware has been uploaded this way already.
For python I use Debian packages wherever possible. What I need is in there usually. I might even say almost always.
Go’s package repository is just GitHub.
At the end of the day, it’s all a URL.
You’re asking for a blessed set of URLs. You’d have to convince someone to spend time maintaining that.
As hair splitting, that's actually not true: Go's package manager is just version control of which GitHub is currently the most popular hosting. And it also allows redirecting to your own version control via `go mod edit -replace` which leaves the sourcecode reference to GitHub intact, but will install it from wherever you like
How does that relate to the bigger conversation here? Are you suggesting people stop pulling Go packages from GitHub and only use local dependencies?
1 reply →
Golang at least gives you the option to easily vendor-ize packages to your local repository. Given what has happened here, maybe we should start doing this more!
This doesn't really help you. I assume Go records the sha1 hash of the commit it grabs, so it doesn't really matter if you vendor it, or download it every time.
The problem comes when you want to upgrade your dependencies. How do you know that they are trustworthy on first use?
1 reply →
npm has always downloaded to the current directory.
1 reply →
The problem with your idea is that you need to find the person who wants to do all this auditing of every version of Node/Python/Ruby libraries.
I believe good centralized infrastructure for this would be a good start. It could be "gamified" and reviewers could earn reputation for reviewing packages, common packages would be reviewed all the time.
Kinda like Stackoverflow for reviews, with optional identification and such.
And honestly an LLM can strap a "probably good" badge on things with cheap batch inference.
Decentralised auditing is what is needed.
https://github.com/crev-dev/
> suffer from this problem
Benefit from this feature.