Comment by bilalq
4 days ago
You're probably already planning this, but please setup an alarm to fire off if a new package release is published that is not correlated with a CI/CD run.
4 days ago
You're probably already planning this, but please setup an alarm to fire off if a new package release is published that is not correlated with a CI/CD run.
Or require manual intervention to publish a new package. I'm not sure why we need to have a fully automated pipeline here to go from CI/CD to public package release. It seems like having some kind of manual user interaction to push a new version of a library would be a good thing.
The basic issue with manual interaction is a question of authority: a pretty common problem for companies (and open source groups) is when $EARLY_EMPLOYEE/$EARLY_CONTRIBUTOR creates and owns the entire publishing process for a key package, and then leaves without performing a proper transfer of responsibility. This essentially locks the company/group out of its own work, and increases support load on community maintained indices to essentially adjudicate rightful ownership of the package name.
(There are a variety of ways to solve this, but the one I like best is automated publishing a la Trusted Publishing with environment mediated manual signoffs. GitHub and other CI/CD providers enable this.)
I don’t buy this. I mean, I’m sympathetic to the issue. It’s easy for things to get jumbled when companies are young. But we’re talking about libraries that are used by your customers. To many, this is the externally visible interface to the company.
What you describe sounds like a process problem to me. If an $EARLY_EMPLOYEE if the only one with the deploy keys for what is a product of the company, then that’s a problem. If a deployment of that key library can be made without anyone approving it, that’s also a problem. But those are both people problems… and you can’t solve a people problem with a technical solution.
6 replies →
You can't "require" manual intervention. Sure you can say that the keys stays on say 2 developers laptops, but personal devices have even more surface area for key leak than CI/CD pipeline. It wouldn't have prevented attacks like this issue in any case where the binary just searched for keys across the system.
One alternative is to do the signing on airlocked system stored in physically safe but accessible location, but I guess that's just way too much inconvenience.
As someone else mentioned, the easiest way would be to have some kind of MFA in the loop. It’s not perfect, but better than what we have now.
I get that it can be useful sometimes. But requiring physical MFA to make a package available to the general public seems like a no-brainer to me.
Users who really want to could opt in to the bleeding edge.
This is orthogonal to the issue at hand. The problem is a malicious actor cutting a release outside of the normal release process. It doesn't matter if the normal process is automated or manual.
It could have eliminated an attack surface where they steal the credentials from the CI/CD...
...But then you if I understand NPM publishing well, you would still have the credentials on someone's computer laying around? I guess you could always revoke the tokens after publishing? It's all balancing convenience and security, with some options being bad at both?
This is built in NPM. You can get an email on every pkg publishing.
Sure, it might be a little bit of noise, but if you get a notice @ 3am of an unexpected publishing, you can jump on unpublishing it.
Very nice way of putting it, kudos!