← Back to context

Comment by woodruffw

24 days ago

There’s a recurrent pattern with these package compromises: the attacker exfiltrates credentials during an initial phase, then pivots to the next round of packages using those credentials. That’s how we saw them make the Trivy to LiteLLM leap (with a 5 day gap), and it’ll almost certainly be similar in this case.

The solution to this is twofold, and is already implemented in the primary ecosystems being targeted (Python and JS): packagers should use Trusted Publishing to eliminate the need for long lived release credentials, and downstreams should use cooldowns to give security researchers time to identify and quarantine attacks.

(Security is a moving target, and neither of these techniques is going to work indefinitely without new techniques added to the mix. But they would be effective against the current problems we’re seeing.)

In this case, the author's NPM account was taken over, email address changed to one the attacker controls, and the package was manually published.

Since the attacker had full control of the NPM account, it is game over - the attacker can login to NPM and could, if they wanted, configure Trusted Publishing on any repo they control.

Axios IS using trusted publishing, but that didn't do anything to prevent the attack since the entire NPM account was taken over and config can be modified to allow publishing using a token.

  • Yeah, NPM should be enforcing 2FA and likely phishing resistant 2FA for some packages/ this should be a real control, issuing public audit events for email address changes, and publish events should include information how it was published (trusted publishing, manual publish, etc).

  • One wonders if Microsoft/npm.js should allow new packages to be published immediately following an account email address change? I mean changes to email address are already recognized as potential attack vectors, so emails are sent to the old address warning of potential account take over. But this seems to have been done at night, so the warning email would not be seen yet. Even so a new package could be published and served to the world immediately. Unless I misunderstand something about the facts this would indicate an extreme lack of imagination in the people at Microsoft who already went through several cycles of hardening the service against supply chain poisoning attacks.

  • Well, that sucks! It’ll be interesting to learn how they obtained a valid second factor or 2FA bypass; that will inform the next round of defenses here.

There's another element to the solution here: runtime behavioral analysis. No matter how completely the maintainer's credentials are compromised, no matter how well the malware is concealed, it still has to act like malware (in the case of LiteLLM, credential harvesting, in this case a remote-access Trojan). It's possible to detect the behavior, rather than relying on supply-chain integrity.

We built a free tool that runs local behavior analysis on your machine, it's caught every supply-chain attack in the last couple weeks: https://www.producthunt.com/products/axios-litellm-detector

There are solutions, the problem is almost always discipline.

  • I don’t know what this means. Discipline is good, but I think you need to have good tools/primitives in place to help people exercise discipline.

    (The classic example being passwords: we wouldn’t need MFA is everybody just “got good” and used strong/unique passwords everywhere. But that’s manifestly unrealistic, so instead we use our discipline budget on getting people to use password managers and phishing-resistant MFA.)

    • Really? You don't know the difference between having a door lock, and using it?

      MFA is typically enforced by organizations, forcing discipline. Individual usage of MFA is dramatically lower