Shai-Hulud Returns: Over 300 NPM Packages Infected

> I’m not sure what sort of Come to Jesus they had to realize their giant pile of sins needed some redemption but they’re trying.

If they were trying, they'd stop doubling down on sunk costs and instead publicly concede that lock files and how npm-the-tool uses them to attempt to ensure the integrity of packages fetched from npm-the-registry is just a poor substitute for content-based addressing that ye olde DVCS would otherwise be doing when told to fetch designated shared objects from the code repo—to be accompanied by a formal deprecation of npm-install for use in build pipelines, i.e. all the associated user guides and documentation and everything else pushing it as best practice.

npm-install has exactly one good use case: probing the registry to look up a package by name to be fetched by the author (not collaborators or people downstream who are repackaging e.g. for a particular distribution) at the time of development (i.e. neither run time nor build time but at the time that author is introducing the dependency into their codebase). Every aspect of version control should otherwise be left up to the underlying SCM/VCS.

> cannot detect premature EOF during the file transfer. It keeps the incomplete file in the cache where the sha hash fails until you wipe your entire cache.

I wonder what circumstances led to saying “this is okay we’ll ship it like that”

but this stuff is basically solved. We have enough history with languages and distribution of packages, repositories, linux, public trust, signing, maintainers, etc.

One key shift is there is no packager anymore. Its just - trust the publisher.

Any language as big as Node should hire a handful of old unix wizards to teach them the way the truth and the life.

[dead]

I would love to use this (for homelab stuff currently) but I would love a way to have vault/openbao be fully configuration-as-code and version controlled, and only have the actual secret values (those that would not be dynamic) in persistent storage.

Definitely curious if you've come up with a way to give each build a short lived vault approle somehow in any CI system.

We do that in our github runners with oidc integration. Works well

You'd be surprised how many people run 'npm i' in their CI. I've seen this on multiple occasions.

'npm ci' is some mitigation, but doesn't protect against getting hit when running 'npm i(nstall)' during development.

”npm install” does not auto-upgrade packages, it installs them according to the lockfile.

It didn’t always work like that, but it has worked like that for many years.

> Pip does not support this; with uv, use `--exclude-newer`. This appears to require a timestamp; so if you always want things up to X days old you'll have to recalculate.

I do this by having my shell init do this:

    export UV_EXCLUDE_NEWER=$(date -Iu -d "14 days ago")

That’s easy to override if you need to but otherwise seamless.

That's good practice. God knows how many times I've been bitten by npm packages breaking on minor or even patch version changes, even when proudly proclaiming to use semver

I'm struggling to understand why Trusted Publishers is any better.

Let's say you have a limited life, package specific scoped, IP CIDR bound publishing key, running on a private GH workflow runner. That key only exists in a trusted clouds secret store (e.g. no one will have access it from their laptop).

Now let's say you're a "trusted" publisher, running on a specific GitHub workflow, and GitHub Org, that has been configured with OIDC on the NPM side. By virtue of simply existing in that workflow, you're now a NPM publisher (run any publish commands you like). No need to have a secret passed into your workflow scope.

If someone is taking over GitHub CI/CD workflows by running `npm i` at the start of their workflow, how does the "Trusted Publisher" find themselves any more secure than the secure, very limited scope token?

A whole single supported CI partner outside their own corporate family. They really planned this out well.

Good point, but until many popular packages stop requiring install.sh to operate, you'll still need to allowlist some of them. That is built into the PNPM tooling, luckily :)

You're right. PNPM disables all install scripts by default. I was just noting one example.

Did you try putting it in your global config file?

Windows: ~/AppData/Local/pnpm/config/rc

macOS: ~/Library/Preferences/pnpm/rc

Linux: ~/.config/pnpm/rc

I think it's a `pnpm-workspace.yaml` setting, for now, but PNPM has been pretty aggressive with expanding this feature set [1].

[1] https://pnpm.io/supply-chain-security

This is like saying "use MacOS and you won't get viruses" in the 2000s

yes bun does both of the things mentioned in the parent comment:

> Unlike other npm clients, Bun does not execute arbitrary lifecycle scripts like postinstall for installed dependencies. Executing arbitrary scripts represents a potential security risk.

https://bun.com/docs/pm/cli/install#lifecycle-scripts

> To protect against supply chain attacks where malicious packages are quickly published, you can configure a minimum age requirement for npm packages. Package versions published more recently than the specified threshold (in seconds) will be filtered out during installation.

https://bun.com/docs/pm/cli/install#minimum-release-age

You don't need post-install scripts for this. Use optionalDependencies instead https://github.com/nrwl/nx/blob/master/packages/nx/package.j...

Each of those deps contains a constraint installing only for the relevant platform.

As stated, you manually approve them.

They didn't deploy the code. That's not how this exploit works. They _downloaded_ the code to their machine. And npm's behavior is to implicitly run arbitrary code as part of the download - including, in this case, a script to harvest credentials and propagate the worm. That part has everything to do with npm behavior and nothing to do with how much anybody reviewed 3P deps. For all we know they downloaded the new version of the affected package to review it!

wait, I short-circuited here. wasn't the very concept of "libraries" created to *not* have to think about what exactly the code does?

imagine reviewing every React update. yes, some do that (Obsidian claims to review every dependency, whether new or an update), but that's due to flaws of the ecosystem.

take a look at Maven Central. it's harder to get into, but that's the price of security. you have to verify the namespace so that no one will publish under e.g. `io.gitlab.bpavuk.` namespace unless they have access to the `bpavuk` GitLab group or user, or `org.jetbrains.` unless they prove the ownership of the jetbrains.com domain.

Go is also nice in that regard - you are depending on Git repositories directly, so you have to hijack into the Git repo permissions and spoil the source code there.

“Personally, I never wear a seatbelt because all drivers on the road should just follow the road rules instead and drive carefully.”

I don’t control all the drivers on the road, and a company can’t magically turn all employees into perfect developers. Get off your high horse and accept practical solutions.

The arguments for doing frequent releases partially apply to upgrading dependencies. Upgrading gets harder the longer you put it off. It’s better to do it on a regular schedule, so there are fewer changes at once and it preserves knowledge about how to do it.

A cooldown is a good idea, though.

There is a Goldilocks effect. Dependency just came out a few minutes ago? There is no time for the community to catch the vulnerability, no real coverage from dependency scans, and it's a risk. Dependency came out a few months ago? It likely has a large number of known vulns

That is indeed what one should do IMO. We've known for a long time now in the ops world that keeping versions stable is a good way to reduce issues, and it seems to me that the same principle applies quite well to software dev. I've never found the "but then upgrading is more of a pain" argument to be persuasive, as it seems to be equally a pain to upgrade whether you do it once every six months or once every six years.

> Why not take it further and not update dependencies at all until you need to because of some missing feature or systems compatibility you need? If it works it works.

Indeed there are people doing that and communities with a consensus such approach makes sense, or at least is not frowned upon. (Hi, Gophers)

This works until you consider regular security vulnerability patching (which we have compliance/contractual obligations for).

Because updates don't just include new features but also bug and security fixes. As always, it probably depends on the context how relevant this is to you. I agree that cooldown is a good idea though.

CI fights this. But that’s peanuts compared to feature branches and nothing compared to lack of a monolith.

We had so many distinct packages on my last project that I had to massively upgrade a tool a coworker started to track the dependency tree so people stopped being afraid of the release process.

I could not think of any way to make lock files not be the absolute worst thing about our entire dev and release process, so the handful of deployables had a lockfile each that was only utilized to do hotfix releases without changing the dep tree out from underneath us. Artifactory helps only a little here.

Just make sure to update when new CVEs are revealed.

Also, some software are always buggy and every version is a mixed bag of new features, bugs and regressions. It could be due to the complexity of the problem the software is trying to solve, or because it's just not written well.

Because if you're too far behind, when you "need" takes days instead of hours.

Because AppSec requires us to adhere to strict vulnerability SLA guidelines and that's further reinforced by similar demands from our customers.

I don't think that this Kantian argument is relevant in tech. We've had LTS versions of software for decades and it's not like every single person in the industry is just waiting for code to hit LTS before trying it. There are a lot of people and (mostly smaller) companies who pride themselves on being close to the "bleeding edge", where they're participating more fully in discovering issues and steering the direction.

The assumption in the post is that scanners are effective at detecting attacks within the cooldown period, not that end-device exploitation is necessary for detection.

(This may end up not being true, in which case a lot of people are paying security vendors a lot of money to essentially regurgitate vulnerability feeds at them.)

To find a vulnerability, one does not necessarily deploy a vulnerable version to prod. It would be wise to run a separate CI job that tries to upgrade to the latest versions of everything, run tests, watch network traffic, and otherwise look for suspicions activity. This can be done relatively economically, and the responsibility could be reasonably distributed across the community of users.

It does scale against this form of attack. This attack propagates by injecting itself into the packages you host. If you pull only 7d after release you are infected 7d later. If your customers then also only pull 7d later they are pulling 14d after the attack has launched, giving defenders a much longer window by slowing down the propagation of the worm.

That worried me too, a sort of inverse tragedy of the commons. I'll use a weeklong cooldown, _someone else_ will find the issue...

Until no-one does, for a week. To stretch the original metaphor, instead of an overgrazed pasture, we grow a communally untended thicket which may or may not have snakes when we finally enter.

There are companies like Helix Guard scanning registries. They advertise static analysis / LLM analysis, but honeypot instances can also install packages & detect certain files like cloud configs being accessed

Your line of reasoning only makes sense if literally almost all developers in the world adopt cooldowns, and adopt the same cooldown.

That would be a level of mass participation yet unseen by mankind (in anything, much less something as subjective as software development). I think we're fine.

For zero/one days, the trick is that you'd pair dependency cooldowns with automatic scanning for vulnerable dependencies.

And in the cases where you have vulnerable dependencies, you'd force update them before the cooldown period had expired, while leaving everything else you can in place.

The docs list this caveat:

> Note that previous stable versions will not be suggested. The package will be completely ignored if its latest published version is within the cooldown period.

Seems like a big drawback to this approach.

Or require manual intervention to publish a new package. I'm not sure why we need to have a fully automated pipeline here to go from CI/CD to public package release. It seems like having some kind of manual user interaction to push a new version of a library would be a good thing.

This is built in NPM. You can get an email on every pkg publishing.

Sure, it might be a little bit of noise, but if you get a notice @ 3am of an unexpected publishing, you can jump on unpublishing it.

Very nice way of putting it, kudos!

No, just the host that was running the package (the exploit was pretty generic and not targeted at PostHog specifically). In fact, so far we think there were 0 production deployments of PostHog because the package was only live for a little bit.

The packages were published using a compromised key directly, not through our ci/cd. We rolled the key, and published a new clean version from our repo through our CI/CD: https://github.com/PostHog/posthog-js/actions/runs/196303581...

As a user of Posthog, this statement is absurd: > Or safer again not to use software this vulnerable.

Nearly all software you use is susceptible to vulnerabilities, whether it's malicious or enterprise taking away your rights. It's in bad taste to make a comment about "not using software this vulnerable" when the issue was widespread in the ecosystem and the vendor is already being transparent about it. The alternative is you shame them into not sharing this information, and we're all worse for it.

Popularity and vulnerability go hand in hand though. You could be pretty safe by only using packages with zero stars on GitHub, but would you be happy or productive?

I've worried about this for a while with Rust packages. The total size of a "big" Rust project's dependency graph is pretty similar to a lot of JS projects. E.g. Tauri, last I checked, introduces about 600 dependencies just on its own.

Like another commenter said, I do think it's partially just because dependency management is so easy in Rust compared to e.g. C or C++, but I also suspect that it has to do with the size of the standard library. Rust and JS are both famous for having minimal standard libraries, and what do you know, they tend to have crazy-deep dependency graphs. On the other hand, Python is famous for being "batteries included", and if you look at Python project dependency graphs, they're much less crazy than JS or Rust. E.g. even a higher-level framework like FastAPI, that itself depends on lower-level frameworks, has only a dozen or so dependencies. A Python app that I maintain for work, which has over 20 top-level dependencies, only expands to ~100 once those 20 are fully resolved. I really think a lot of it comes down to the standard library backstopping the most common things that everybody needs.

So maybe it would improve the situation to just expand the standard library a bit? Maybe this would be hiding the problem more than solving it, since all that code would still have to be maintained and would still be vulnerable to getting pwned, but other languages manage somehow.

I agree partly. I love cargo and can’t understand why certain things like package namespaces and proof of ownership isn’t added at a minimum. I was mega annoyed when I had to move all our Java packages from jcenter, which was a mega easy setup and forget affair, to maven central. There I suddenly needed to register a group name (namespace mostly reverse domain) and proof that with a DNS entry. Then all packages have to be signed etc. In the end it was for this time way ahead. I know that these measures won’t help for all cases. But the fact that at least on npm it was possible that someone else grabs a package ID after an author pulled its packages is kind of alarming. Dependency confusion attacks are still possible on cargo because the whole - vs _ as delimiter wasn’t settled in the beginning. But I don’t want to go away from package managers or easy to use/sharable packages either.

I'm a huge Go proponent but I don't know if I can see much about Go's module system which would really prevent supply-chain attacks in practice. The Go maintainers point [1] at the strong dependency pinning approach, the sumdb system and the module proxy as mitigations, and yes, those are good. However, I can't see what those features do to defend against an attack vector that we have certainly seen elsewhere: project gets compromised, releases a malicious version, and then everyone picks it up when they next run `go get -u ./...` without doing any further checking. Which I would say is the workflow for a good chunk of actual users.

The lack of package install hooks does feel somewhat effective, but what's really to stop an attacker putting their malicious code in `func init() {}`? Compromising a popular and important project in this way would likely be noticed pretty quickly. But compromising something widely-used but boring? I feel like attackers would get away with that for a period of time that could be weeks.

This isn't really a criticism of Go so much as an observation that depending on random strangers for code (and code updates) is fundamentally risky. Anyone got any good strategies for enforcing dependency cooldown?

[1] https://go.dev/blog/supply-chain

Historically, arguments of "it's popular so that's why it's attacked" have not held up. Notable among them was addressing Windows desktop security vulnerabilities. As Linux and Mac machines became more popular, not to mention Android, the security vulnerabilities in those burgeoning platforms never manifested to the extent that they were in Windows. Nor does cargo or pip seem to be infected with these problems to the extent that npm is.

> It's not "node" or "Javascript" the problem, it's this convenient packaging model.

That and the package runtime runs with all the same privileges and capabilities as the thing you're building, which is pretty insane when you think about it. Why should npm know anything outside of the project root even exists, or be given the full set of environment variables without so much as a deny list, let alone an allow list? Of course if such restrictions are available, why limit them to npm?

The real problem is that the security model hasn't moved substantially since 1970. We already have all the tools to make things better, but they're still unportable and cumbersome to use, so hardly anything does.

Every time I look at a new project, my face falls when it's written in Rust. I simply don't trust a system that pulls in gigabytes of god-knows-what off the cloud, and compiles it on my box. It's a real barrier to entry, for me.

When I download a C project, I know that it only depends on my system libraries - which I trust because I trust my distro. Rust seems to expect me to take a leap in the dark, trusting hundreds of packagers and their developers. That might be fine if you're already familiar with the Rust ecosystem, but for someone who just wants to try out a new program - it's intimidating.

I think this is right about Rust and Cargo, but I would say that Rust has a major advantage in that it implements frozen + offline mode really well (which if you use, obviously significantly decreases the risks).

Any time I ever did the equivalent with NPM/node world it was basically unusable or completely impractical

There are ecosystems that have package managers but also well developed first party packages.

In .NET you can cover a lot of use cases simply using Microsoft libraries and even a lot of OSS not directly a part of Microsoft org maintained by Microsoft employees.

Supply chain attacks are scary because you do everything "right", but the ecosystem still compromises you.

But realistically, I think the sum total of compromises via package managers attacks is much smaller than the sum total of compromises caused by people rolling their own libraries in C and C++.

It's hard to separate from C/C++'s lack of memory safety, which causes a lot of attacks, but the fact that code reuse is harder is a real source of vulnerabilities.

Maybe if you're Firefox/Chromium, and you have a huge team and invest massive efforts to be safe, you're better off with the low-dependency model. But for the median project? Rolling your own is much more dangerous than NPM/Cargo.

Rust (and really, any but JS) ecosystem have a bit more "due dilligence" applied everywhere; I don't doubt someone will try to namesquat but chance of success are far smaller

> The more I think about it, the more I believe that C, C++ or Odin's decision not to have a convenient package manager that fosters a cambrian explosion of dependencies to be a very good idea security-wise.

There was no decision in case of C/C++; it was just not a thing languages had at the time so the language itself (especially C) isn't written in a way to accommodate it nicely

> Ambivalent about Go: they have a semblance of packaging system, but nothing so reckless like allowing third-party tarballs uploaded in the cloud to effectively run code on the dev's machine.

Any code you download and compile is running code on dev machine; and Go does have tools to do that in compile process too.

I do however like the by default namespacing by domain, there is no central repository to compromise, and forks of any defunct libs are easier to manage.

I have a similar opinion but I think Java's model with maven and friends hits the sweet spot:

- Packages are always namespaced, so typosquating is harder - Registries like Sonatype require you to validate your domain - Versions are usually locked by default

My professional life has been tied to JVM languages, though, so I might be a bit biased.

I get that there are some issues with the model, especially when it comes to eviction, but it has been "good enough" for me.

Curious on what other people think about it.

While I agree that dependency tree size can be sometimes a problem in Rust, I think it often gets overblown. Sure, having hundreds of dependencies in a "simple" project can be scary, but:

1) No one forces you to use dependencies with large number of transitive dependencies. For example, feel free to use `ureq` instead of `reqwest` pulling the async kitchen sink with it. If you see an unnecessary dependency, you could also ask maintainers to potentially remove it.

2) Are you sure that your project is as simple as you think?

3) What matters is not number of dependencies, but number of groups who maintain them.

On the last point, if your dependency tree has 20 dependencies maintained by the Rust lang team (such as `serde` or `libc`), your supply chain risks are not multiplied by 20, they stay at one and almost the same as using just `std`.

> The more I think about it, the more I believe that C, C++ or Odin's decision not to have a convenient package manager that fosters a cambrian explosion of dependencies to be a very good idea security-wise.

The safest code is the code that is not run. There is no lack of attacks targeting C/C++ code, and odin is just a hobby language for now.

Don't worry about C or C++, we create the vulnerabilities ourselves !

Using C++ daily, whenever I do js/ts are some javascript variant, since I don't use it daily, and update becomes a very complex task. frameworks and deps change APIs very frequently.

It's also very confusing (and I think those attack vectors benefit exactly from that), since you have a dependency but the dep itself dependent on another dep version.

Building basic CapacitorJS / Svelte app as an example, results many deps.

It might be a newbie question, but, Is there any solution or workflow where you don't end up with this dependency hell?

Not knowing that much about apt, isn't _any_ package system vulnerable, and purely a question of what guards are in place and what rights are software given upon install?

Agreed with the first half, but giving up on convenient packaging isn't the answer.

Things like cargo-vet help as does enforcing non-token auth, scanning and required cooldown periods.

Out of the 789 npm packages in this incident, only 4 were ever used in any dependency tree of any Linux operating system (including Homebrew). Not in the affected versions, but ever.

If your Rust software observes a big enough chunk of the computer fever dream you are likely to end up with 2-3 digit amount of Rust dependencies, but they are probably all going to be high profile ones (tokio, anyhow, reqwest, the hyper crates, ...), instead of niche ones that never make it into any operating system.

This is not a silver bullet of course, but there seems to be an inverse correlation between "is part of any operating system dependency tree" and "gets compromised in an npm-like incident".

> The more I think about it, the more I believe that C, C++ or Odin's decision not to have a convenient package manager that fosters a cambrian explosion of dependencies to be a very good idea security-wise. Ambivalent about Go: they have a semblance of packaging system, but nothing so reckless like allowing third-party tarballs uploaded in the cloud to effectively run code on the dev's machine.

The alternative that C/C++/Java end up with is that each and every project brings in their own Util, StringUtil, Helper or whatever class that acts as a "de-facto" standard library. I personally had the misfortune of having to deal with MySQL [1], Commons [2], Spring [3] and indirectly also ATG's [4] variants. One particularly unpleasant project I came across utilized all four of them, on top of the project's own "Utils" class that got copy-and-paste'd from the last project and extended for this project's needs.

And of course each of these Utils classes has their own semantics, their own methods, their own edge cases and, for the "organically grown" domestic class that barely had tests, bugs.

So it's either a billion "small gear" packages with dependency hell and supply chain issues, or it's an amalgamation of many many different "big gear" libraries that make updating them truly a hell on its own.

[1] https://jar-download.com/artifacts/mysql/mysql-connector-jav...

[2] https://commons.apache.org/proper/commons-lang/apidocs/org/a...

[3] https://docs.spring.io/spring-framework/docs/current/javadoc...

[4] https://docs.oracle.com/cd/E55783_02/Platform.11-2/apidoc/at...

My feeling is that languages with other packing models are merely less convenient, and there is no actual tangible difference security-wise. Just take C and replace "look for writable repositories". It just takes more work and is less uniform to say write a worm that looks for writable cmake/autoconf and replicate that way.

What would actually stop this is writing compilers and build systems in a way that isolates builds from one another. It's kind of stupid that all a compiler really needs is an input file, a list of dependencies, and an output file. Yet they all make it easy to root around, replicate and exfiltrate. It can be both convenient and not suffer from these style of attacks.

Not having a convenient package manager doesn't mean you don't need the functionality that's otherwise offered by third-party packages, it just means that you either need other means to obtain those third-party packages (usually reducing the visibility this dependency!) or implement them yourself (sometimes this is good, but sometimes this can also be very bad for security. Your DYI code won't get as many eyes and audits as the popular third party package!).

Must read: https://wiki.alopex.li/LetsBeRealAboutDependencies

Indeed, Rust's supply chains story is an absolute horror, and there are countless articles explaining what should be done instead (e.g. https://kerkour.com/rust-stdx)

TL;DR: ditch crates.io and copy Go with decentralized packages based directly on and an extended standard library.

Centralized package managers only add a layer of obfuscation that attackers can use to their advantage.

On the other hand, C / C++ style dependency management is even worse than Rust's... Both in terms of development velocity and dependencies that never get updated.

I believe you, in that package management with dependencies without security mitigation is both convenient and dangerous. And I certainly agree this could happen for other package managers as well.

My real worry, for myself re the parent comment is, it's just a web frontend. There are a million other ways to develop it. Sober, cold risk assessment is: should we, or should we have, and should anyone else, choose something npm-based for new development?

Ie not a question about potential risk for other technologies, but a question about risk and impact for this specific technology.

> C/C++ .. a convenient package manager

Every time I fire up "cmake" I chant a little spell that protects me from the goblins that live on the other side of FetchContent to promise to the Gods of the Repo that I will, eventually, review everything to make sure I'm not shipping poop nuggets .. just as soon as I get the build done, tested .. and shipped, of course .. but I never, ever do.

Surely in this case the problem is a technical one, and with more work towards a better security model and practices we can have the best of both worlds, no?

Agreed, rust's cargo model is basically the worst part of that ecosystem right now. I've had developers submit pretty simple cli tools with hundreds and hundreds of dependencies. I guess there wasn't any lessons learned from the state of NPM.

Just a last month someone was trying to figure the cargo tree on which Rust package got imported implicitly via which package. This will totally happen in rust as well as long as you use some kind of package manager. Go for zero or less decencies.

It’ll probably happen eventually with Rust, but ecosystem volume and informal packaging processes / a low barrier to entry seem to be significant driver in the npm world.

(These are arguably good things in other contexts.)

I don’t get this

I installed the package, obviously I intend to run it. How does getting pwned once I run it manually differ from getting pwned once I install it? I’m still getting pwned

In the early days the Node ecosystem adopted (from Unix) the notion that everything has to be its own micro package. Not only was there a failure to understand what it was actually talking about, but it was never a good fit for package management to begin with.

I understand that there's been some course correction recently (zero dependency and minimal dependency libs), but there are still many devs who think that the only answer to their problem is another package, or that they have to split a perfectly fine package into five more. You don't find this pattern of behavior outside of Node.

Node is the embodiment of move and break things. Probably will not build anything that should last more than a few months on node.

Why the word "semblance" with regard to Go modules? Are you trying to say this system is lacking something?

An open question is why PyPI doesn’t have the same problem.

maybe the solution is what linux & co used for many years: have a team of people who vet and package dependencies.

do they follow the same process ? or is it harder to submit a package and vet it on rust/cargo ?

I hate to be the guy saying AI will solve it, but this is a case where AI can help. I think in the next couple of years we’ll see people writing small functions with Claude/codex/whatever instead of pulling in a dependency. We might or might not like the quality of software we see, but it will be more resistant to supply chain attacks.

Go is just as bad.

> but it's only a matter of time until it'll happen on the Rust ecosystem

Totally 100% agree, though tools like cargo tree make it more of a tractable problem, and running vendored dependencies is first class at least.

The one I am genuinely most concerned of is Golang. The way Dependencies are handled leaves much to be desired, I'm really surprised that there haven't been issues honestly.

NPM has about 4 million packages, Maven Central has about 3 million packages.

If this were true, wouldn't there have been at least one Maven attack by now, considering the number of NPM attacks that we've seen?

The concern is not 'could' happen, but _does_ happen. I know this could occur in many places. But where it seems highly prevalent is NPM.

And I am genuinely thinking to myself, is this making using npm a risk?

Value is one thing but the average user (by virtue of being popular) will be just less clued in on any security practices that could mitigate the problem.

Microsoft owns npmjs.com. They could pay for AI analysis of published version deltas, looking for backdoors and malware.

It has everything to do with node/js. Because the community believes in tiny dependencies that must be updated as often as possible and the tooling reflects that belief.

it's interesting that staying up to date with your dependencies is considered a vulnerability in Node

Not sure what the language has anything to do with it, we've built JavaScript applications within pulling in 100s of NPM packages before NPM was a thing, people and organizations can still do so today, without having to switch language, if they don't want to.

Does it require disciple and a project not run by developers who just learned program? You betcha.

Ah yes PHP, the language known for its strong security...

Did Phoenix not require npm at some point or is that not true?

Yes, and you can code in assembly as well if you want it. But: that's not how 99% of the people using node is using it so that it is theoretically possible to code up every last bit yourself is true but it does not contribute to the discussion at all.

An eco-system, if it insists on slapping on a package manager (see also: Rust, Go) should always properly evaluate the resulting risks and put proper safeguards in place or you're going to end up with a massive supply chain headache.

So your supposed to write your own posthog? be serious

npm has been the official package manager for node since forever (0.8 or earlier iirc). I think even before the io.js fork and merge.

.NET being so batteries-included is one of its best features. And when vulnerabilities do creep in, it's nice to know that Microsoft will fix it rather than hoping a random open source project will.

This a common refrain on HN, frequently used to dismiss what may be perfectly legitimate concerns.

It also ignores the central question of whether NPM is more vulnerable to these attacks than other package managers, and should therefore be considered an unreasonable security risk.

There are far too many languages and many packages for each of them for this (good) idea to be practicable.

You can write simple front-end without reactive components. Most pages are not full blown apps and they were fine for a very long time with jQuery, whose features have been largely absorbed into plain js/dom/CSS.

Yup! No new technologies have been invented or discovered thru building websites since CSS 1.0 in 1996.

If everyone does it, then it becomes less effective, because there'd be fewer early testers to experience and report issues, no?

You forget to account for the fact that the xz-utils backdoor was extremely high effort. Literally a high skilled person building trust over time. While it's obviously possible and problematic, it's still a scaling/time issue.

Node the technology can be used without blindly relying on the update features of npm. Vet your dependency trees, lock your dependency versions at patch level and use dependency cooldown.

This is something you also need to do with package managers in other languages, mind you.

You need to explain how one is supposed to distinguish and exclude "namespaces pretty much nobody uses" when writing code in this ecosystem. My understanding is that a typical Node developer pretty much has no control over what gets pulled in if they want to get anything done at all. If that's the case, then you don't have an argument. If a developer genuinely has no control, then the point is moot.

The only way a worm like this spreads is usage of the affected packages. The proliferation itself is clear evidence of use.

Ok, I'll bite; which package repos are "the same or worse" than those of nodejs?

I see a bunch under major SaaS vendor namespaces that have millions of weekly downloads…?

It's like seatbelts/car seats.

Everything runs in the container and cannot escape it. Its like a sandbox.

You have to make sure you're not putting any secrets in the container environment.

I didn't read this as separate containers.

What do you mean? You can drop into bash in a container and run any arbitrary command, so `npm install foo` works just fine. Why would posthog's SDK be a special case?

I send mail to a demon which runs MsBuild and mails the output back to me.

Security is oft done in layers. Podman is just another layer that might fail. There's no such thing as the perfect solution.

the whole point of doing such things in ephemeral containers is to not have your secrets accessible from said container

Agreed that some level of gatekeeping and some level of friction to protect quality are useful things.

No. I am tired of seeing people dragged into poor quality environments for the wrong reasons. These people would do much better using other tools.

Your attempt to make it personal does not compute.

Well it certainly doesn't feel good to get your shit pwned every two weeks.

The reality of my life is that I have to write code targeting Node because of choices that other people made for their libraries and products.

I didn't hate JS & npm with a passion before. I do now, and it hasn't even been two years.

Thanks. Also - maybe change "talk to a human" to "talk to a grumpy robot" :)

Your status page isn't clear, but are all versions between the compromised and "safe to install" versions compromised or just the ones listed?

For example I installed `posthog-react-native` version `4.12.4` which is between the `4.11.1` version which is compromised and the safe to install version `4.13.0`. Is that version compromised or not?

This is now the main thread. Though dang likes to merge dupes.

See also Xe Iaso's posts about CVEs in the C ecosystem (https://xeiaso.net/shitposts/no-way-to-prevent-this/CVE-2025...)

I would say that npm likely has easier solutions here compared to Cargo.

Well before the npm attacks were a thing, we within the Rust project, have discussed a lot of using wasm sandboxing for build-time code execution (and also precompiled wasm for procedural macros, but that's its own thing.) However the way build scripts are used in the Rust ecosystem makes it quite difficult enforce sandbox while also enabling packages to build foreign code (C, C++ invoke make, cmake, etc.) The sandbox could still expose methods to e.g. "run the C compiler" to the build scripts, but once that's done they have an arbitrary access to a very non-trivial piece of code running in a privileged environment.

Whereas for Javascript rarely does a package invoke anything but other javascript code during the build time. Introduce a stringent sandbox for that code (kinda deno style perhaps?) and a large majority of the packages are suddenly safe by default.

This is a cultural problem created through a fundamental misunderstanding (and mis-application) of Unix philosophy. As far as I'm aware the Rust ecosystem doesn't have a problem appropriately sizing packages which in turn reduces the overall attack surface of dependencies.

You can host your own NPM reg, and examine every package, but your manager probably is NOT going to go for that.

It's not just the popularity, it's partly the update mechanism and partly the culture. In what sane world would you always pull in all the newest things, regardless or whether you need them or not? This is a default at build time for so many setups. If you absolutely must use that package manager, at least lock down your versions, and update selectively. I don't even know if that's possible to do with the dependencies' dependencies (and so on), or are people forced to just pull in whatever, every time.

An example: Java Maven artifacts typically name the exact version of their dependencies. They rarely write "1.2.3 or any newer version in the 1.2.x series", as is the de-facto standard in NPM dependencies. Therefore, it's up to each dependency-user to validate newer versions of dependencies before publishing a new version of their own package. Lots of manual attention needed, so a slower pace of releases. This is a good thing!

Another example: all Debian packages are published to unstable, but cannot enter testing for at least 2-10 days, and also have to meet a slew of conditions, including that they can be and are built for all supported architectures, and that they don't cause themselves or anything else to become uninstallable. This allows for the most egregious bugs to be spotted before anyone not directly developing Debian starts using it.

Other languages seem to publish dependencies as self-contained packages whose installation does not require running arbitrary shell scripts.

This does not prevent said package from shipping with malware built in, but it does prevent arbitrary shell execution on install and therefore automated worm-like propagation.

You have separate people called "maintainers", and they're the ones who build and upload packages to the repository. Crucially, they're not the people who write the software. You know, like Linux has been doing since forever. https://wiki.debian.org/DebianMaintainer Instead of treating your package repository like a trash can at a music festival, you can treat it more like a museum, curated by experts. Unfortunately, this isn't quite the devil-may-care attitude the Node ecosystem is so accustomed to, and will be met with a lot of whining, so it never happens. See y'all in two weeks when this happens again.

Build packages from source without any binaries (all the way down) and socially audit the source before building.

https://bootstrappable.org/ https://reproducible-builds.org/ https://github.com/crev-dev

The same way it always has been done - vendor your deps.

I think some system would need to dynamically analyze the code (as it runs) and record what it does. Even then, that may not catch all malicious activity. It's sort of hard to define what malicious activity is. Any file read or network conn could, in theory, be malicious.

As a SW developer, you may be able to limit the damage from these attacks by using a MAC (like SELinux or Tomoyo) to ensure that your node app cannot read secrets that it is not intended to read, conns that it should not make, etc. and log attempts to do those things.

You could also reduce your use of external packages. Until slowly, over time you have very little external dependencies.

Other than general security practices, here are few NPM ecosystem specific ones: https://github.com/bodadotsh/npm-security-best-practices

Hire an antivirus company to provide a safe and verified feed of packages. Use ML and automatic scanners to send packages to manual review. While Halting problem prevents us from 100% reliably detecting malware, at least we can block everything suspicious.

I don't get this explanation. How does it force you to run the infection code?

Yes, if you depend on an infected package, sure. But then I'd expect not just a list, but a graph outlining which package infected which other package. Overall I don't understand this at all.

Thanks. I saw that sentence but somehow didn't parse it. Need a coffee :/

No.1: Run untrusted code in a sandbox! https://github.com/sandbox-utils/sandbox-venv

Most of the best practices can be translated to python ecosystem. It’s not exact 1:1 mapping but change few key terms and tools, the underlying practices should be the same.

Or copy that repo’s markdown into an llm and ask it to map to the pip ecosystem

So, I do this because it's universally recommended, but why does it help?

Can't they just jam the malware into the package itself? It runs with the same permissions on my machine (in unit tests, node servers, etc).

> I think it's better to not run npm as root user on container. I would suggest adding --user 1000 to your docker run command.

Good point. Here's the improvement that work for me

https://github.com/ashishb/dotfiles/commit/fe4fb15fe867bf77a...

That solution is not easy, and as stated it takes work. And there is a serious problem of funding people to do that work.

And it has not yet been demonstrated at PyPI/NPM scale, either.

fixed!

Well, writing obfuscated code like ["e" + "v" + "a" + "l"]() is already a huge red flag for sending the package to manual review. While it might be impossible to detect all methods of obfuscation, we could start with known methods.

Also, detecting obfuscated code sounds like an interesting and challenging task.

There's always some mathematician who tries to prove that locks on your doors "won't help" because the universe is infinite. Narrator: it is not

They probably don't willy-nilly install every new patch that comes down the pike?

I agree it's hard. But it's actually easier in professional settings. There are funds and you don't have an excuse to be lazy.

At minimum whatever you are working on should be built in docker. The package installation then would happen during the image build step. Yes it's easy to break out of the isolation environment but i am betting this malware does not.

NPM tokens should exist in some configuration/secret management solution not on your home directory. Devs have no business holding the NPM tokens. Same goes for sensitive environment variables they have no business existing on dev laptops or even the pipeline build steps (where package installation should happen).

AWS etc credentials / tokens are harder to secure since there are legit reasons for existing in dev laptops.

My code editor works in a sandbox. It's difficult because Linux doesn't provide it and one has to write it manually using shell scripts, random utilities. For example, I had also to write a limited FUSE emulation of /proc to allow code editor work without access to real /proc which contains lot of unnecessary information.

And if it's a "professional" setting, the company could hire a part-time developer for writing the sandbox.

Sure it is: don't do it. It's not like there isn't automated tooling for this, but you can also like... I don't know, look at your diffs and not commit secrets? I've never committed a secret before, and I've been working with AWS for ten years now. But I don't "git commit -a" and I triple-check my diffs.

When I worked in commercial aerospace, before we even shipped live there was an incident with a CERT advisory against the XML package we were using. But the fix was only added to the current major version and we were stuck one behind. It took ~3 of our best problem solvers about a week to get that damned thing upgraded. Which put us behind on our schedule.

This made some of my more forward thinking coworkers nervous because what if this happened after we went live? So we started a repeating story called “upgrade dependencies” and assigned it round robin once a month to someone on each application. Every time someone got it the first time they would ask me, “but upgrade what?” Whatever you want, but preferable something that hasn’t been in a while.

For IP and security reasons we were already on vendored dependencies, so it was pretty straightforward to tell what was old. But that made “upgrade immediately” problematic if fixes weren’t back ported far enough and we didn’t want that live.

I was working with the assumption in this model the attestation is signed by ephemeral keys (OIDC) which would reveal the bad actor or give breadcrumbs. Enough to reduce incentives to hijack packages.

Maybe we can convince Shopify to hijack NPM too while they're at it.

You don't even need to enshittify Yet Another Service, you just need package maintainers. Debian manages to do this, and I'm guessing they get paid nothing (although, yeah, Amazon and The Goog really ought to chip in a few bucks, considering their respective empires). Unfortunately, it means you can't just YOLO your code into other people's programs anymore.

Nah - dependency cooldown is all the rage but it’s only effective if you have some noncompliant canary users. Once everyone is using it it will cease to be effective because nobody will be taking the first step/risk until everybody does.

Or bun

99% of releases do NOT fix zero-days. But 100% of releases have a small risk of introducing a backdoored build-script.

There's nothing wrong with pinning dependencies and only updating when you know for sure they're fixing a zero-day (as it will be public at that point).

Or an old enough version. For one of the most damaging zero-day vulnerabilities in the Java ecosystem (log4shell), you were vulnerable if you were in the latest version, but not vulnerable if you were using an old enough version.

Zero-day on frontend has not really a y effect, except on one user at a time. Zero-day on a server though ... perhaps we arrive at the conclusion to not use the JS ecosystem on the server side.

do zero-days even care about versions?

Exactly, there is no easy solution to these problems.

The solutions that are effective also involve actually doing work, as developers, library authors, and package managers. But no, we want as much "convenience" as possible, so the issues will continue.

Developers and package authors should use a lockfile, pin their dependencies, be frugal about adding dependencies, and put any dependencies they do add through a basic inspection at least, checking what dependencies they also use, their code and tests quality, etc.

Package managers should enforce namespacing for ALL packages, should improve their publishing security, and should probably have an opt-in verified program for the most important packages.

Doing these will go a long way to ameliorate these supply chain attacks

How does having a mirror help?

This is true. Today its npm, tomorrow it could be some other language. Shouldnt we focus on solving it at the root?

Some of us need to drink from the river to eat :(

> Including headers isn't remotely "simple". There's so many considerations in linking, .SO version compatibility, architecture and instruction set issues, building against multiple versions on the same system. Or if you want to feel frustrated in a single word: GDAL (IYKYK)

I'd say for issues with instruction sets is a minor edge case, you could argue that having layers of abstraction gives even more room for fault.

That said I am not familiar with GDAL, but from google-fu it seems relatively heavy. Being said, I don't have much familiarity with compiling CXX programs as I do with programming C programs, and C with it's smaller footprint tends to not give me as many problems.

Again, this is from personal experience.

[dead]

Ecosystems aren’t built by any homogeneous group of people. They’re a sum of their parts. It’s not like there was a committee and that committee decided how things should work wrt wheel reinvention. People publish packages, and the result is something we call an ecosystem.

Does it have a terminal assistant that I have not heard of? Otherwise, the parent asks about an assistant that is able to run various tools and stuff, not just talk.

For context: ramada 0.32.0 isn't a concrete thing, in the sense that glibc 2.35 is. It really means "the latest ramada code because if you were to pin on this version it'll at some point stop working". glibc 2.35 never stops working.

> the trade off

I see that as a desirable feature. I don’t want new functionality suddenly popping into my codebase without one of my team intending it.

But you would have code reviewed it

.Net has Blazor for WebUI.

I don't ask you to judge if you like it, I'm just saying that you can totally make a professional WebUI within the dotnet stdlib.

Respectfully disagree. Python and Go std lib do not even play in the same league. I had to help someone with datetime¹ handling in Python a while back. The stdlib is so poor, you have to reach out for a thirdparty lib for even the most basic of tasks².

Don't take my word for it, take a dive. You wouldn't be the first to have adjust their view.

For example, this section is just about the built-in web framework asp.net: https://learn.microsoft.com/en-us/aspnet/core

______

1. This might be a poor example as .net has NodaTime and the jvm has YodaTime as 3rd-party libs, for if one has really strict needs. Still, the builtin DateTime constructs offer way more than what Python had to offer.

2. Don't get me started on the ORM side of things. I know, you don't have to use one, but if you do, it better does a great job. And I wouldn't bat an eye if the ORM is not in the standard, but boy was I disappointed in Python's ecosystem. EF Core come batteries included and is so much better, it isn't fun anymore.

But they could encrypt it, its just double b64 encoded, everybody can read it.

What is genuine sandboxing? Everyone waives there hands by saying this

Second screen experience

Why? Everyone won't use cooldowns, but the key is to have just enough people running brand new to set off a warning/have systems that check dependencies scan and find vulns go off and the packages get pulled before production builds them.

Monocultures where everyone pulls and builds with every brand new thing for the most minor changes is dangerous.