Comment by hinkley
4 days ago
Npm is what happens when you let tech debt stack up for years too far. It took them five attempts to get lock files to actually behave the way lock files are supposed to behave (lockfile version 3, + at least 2 unversioned attempts before that).
It’s clear from the structure and commit history they’ve been working their asses off to make it better, but when you’re standing at the bottom of a well of suck it takes that much work just to see daylight.
The last time I chimed in on this I hypothesized that there must have been a change in management on the npm team but someone countered that several of the maintainers were the originals. So I’m not sure what sort of Come to Jesus they had to realize their giant pile of sins needed some redemption but they’re trying. There’s just too much stupid there to make it easy.
I’m pretty sure it still cannot detect premature EOF during the file transfer. It keeps the incomplete file in the cache where the sha hash fails until you wipe your entire cache. Which means people with shit internet connections and large projects basically waste hours several times a week doing updates that fail.
> I’m not sure what sort of Come to Jesus they had to realize their giant pile of sins needed some redemption but they’re trying.
If they were trying, they'd stop doubling down on sunk costs and instead publicly concede that lock files and how npm-the-tool uses them to attempt to ensure the integrity of packages fetched from npm-the-registry is just a poor substitute for content-based addressing that ye olde DVCS would otherwise be doing when told to fetch designated shared objects from the code repo—to be accompanied by a formal deprecation of npm-install for use in build pipelines, i.e. all the associated user guides and documentation and everything else pushing it as best practice.
npm-install has exactly one good use case: probing the registry to look up a package by name to be fetched by the author (not collaborators or people downstream who are repackaging e.g. for a particular distribution) at the time of development (i.e. neither run time nor build time but at the time that author is introducing the dependency into their codebase). Every aspect of version control should otherwise be left up to the underlying SCM/VCS.
> cannot detect premature EOF during the file transfer. It keeps the incomplete file in the cache where the sha hash fails until you wipe your entire cache.
I wonder what circumstances led to saying “this is okay we’ll ship it like that”
I think we can blame the IO streaming API in NodeJS on this. It’s a callback and you just know you got another block. My guess is chunked mode and not checking whether the bytes expected and the bytes received matched.
Not to diminish the facepalm but the blame can be at least partially shared.
Our UI lead was getting the worst of this during Covid. I set up an nginx forward proxy mostly for him to tone this down a notch (fixed a separate issue but helped here a bit as well) so he could get work done on his shitty ISP.
Ignorance. Most programmers in open source operate on the "works on my machine"
True, and things that manifest only on old/slow hardware or on bad internet are the worst kind for this, since 100% of developers who have any say in the matter would never accept such circumstances at all, so they’re always approaching every issue with multi-gigabit speeds, zero latency, and this year’s $3,000 Mac. “What do you mean the page loads slowly?”
1 reply →
but this stuff is basically solved. We have enough history with languages and distribution of packages, repositories, linux, public trust, signing, maintainers, etc.
One key shift is there is no packager anymore. Its just - trust the publisher.
Any language as big as Node should hire a handful of old unix wizards to teach them the way the truth and the life.
Likely they wouldn’t listen. Modern languages and environments seem intent on reinventing bad solutions to solved problems. I get it if it’s a bunch of kids that have never seen anything better but there is no excuse these days not to have at least a passing knowledge of older systems if you’ve been around a while.
there's certainly a piece of it. Also most seasoned people are not very interested in new languages and environments, and most languages are not 'spec built' by experts like Rob Pike building Go who explicitly set out to solve a lot of his problems, but are more naturally grown and born.
> One key shift is there is no packager anymore. Its just - trust the publisher.
Repositories like NPM's, and PyPI, contain many more packages than any Linux distro. And the Linux Foundation actually gets funded.
NPM isn't a package repository it's more akin a code repository.
There's a reason why most distributions dont ship upstream (except basically Arch)
[dead]