Comment by PaulHoule
7 hours ago
Would be nice if node packages could be packed up in ZIP files so to avoid the security/metadata tax for small file access on Windows.
7 hours ago
Would be nice if node packages could be packed up in ZIP files so to avoid the security/metadata tax for small file access on Windows.
The number of files in the node modules folder is crazy, any amount of organization that can tame that chaos is welcomed.
And if you thought malware hiding in a mess of files was bad, just wait till you see it in two layers of container files.
Or worse yet, the performance load of anti-malware software that has to look inside ZIP files.
Look, most of us realized around 2004 or so that if you had a choice between Norton and the virus you would pick the virus. In the Windows world we standardized around Defender because there is some bound on how much Defender degrades the performance of your machine which was not the case with competitive antivirus software.
I've done a few projects which involved getting container file formats like ZIP and PDF (e.g. you know it's a graph of resources in which some of those resources are containers that contain more resources, right?) and now that I think of it you ought to be able to virus scan ZIP files quickly and intelligently but the whole problem with the antivirus industry is that nobody ever considers the cost.
1 reply →
There are alternative package managers like Yarn that use zip files as a way to store each Node package.[0]
[0] https://yarnpkg.com/advanced/pnp-spec#zip-access
Strong recommendation to use PNPM instead of yarn or npm. IME (webdev since 1998) it's the only sane tool for stewardship of an npm dependency graph.
See https://pnpm.io/motivation
Also, while popularity isn't necessarily a great indicator of quality, a quick comparison shows that the community has decided on pnpm:
https://www.npmcharts.com/compare/pnpm,yarn,npm
yarn with zero-installs removes an awful lot of pain present in npm and pnpm. Its practically the whole point of yarn berry.
Firstly - with yarn pnp zero-installs, you don't have to run an `install` every time you switch branch, just in case a dep changed. So much dev time is wasted due to this.
Secondly - "it worked on my machine" is eliminated. CI and deploy use the exact same files - this is particularly important for deeply nested range satisfied dependencies.
Thirdly - packages committed to the repo allows for meaningful retrospectives and automated security reviews. When working in ops, packages changing is hell.
All of this is facilitated by the zip files that the comment you replied to was discussing, that you tangented away from.
The graph you have linked is fundamentally odd. Firstly - there is no good explanation of what it is actually showing. I've had claude spin on it and it reckons its npm download counts. This leads to it being a completely flawed graph! Yarn berry is typically installed either via corepack or bootstrapped via package.json and the system yarn binary. Yarn even saves itself into your repo. pnpm is never (I believe) bundled with the system node, wheras yarn and npm typically are.
Your graph doesn't show what you claim it does.
... and of course JAR files in Java are just ZIP files with a little extra metadata and the JVM can unpack them in realtime just fine.
When npm decided to have per-project node_modules (rather than shared like ruby and others) and human readable configs and library files I think the goal was to be a developer friendly and highly configurable, which it is. And package.json became a lot more than that as a result, it’s been a great system IMO.
Combined with a hackable IDE like Atom (Pulsar) made with the same tech it’s a pretty great dev exp for web devs
I remember when Firefox started putting everything into jars for similar reasons.
https://web.archive.org/web/20161003115800/https://blog.mozi...
Would accessing deps directly from a zip really be faster? I'd be a little surprised but not terribly, given that it's readonly on an fs designed for RW. If not, maybe just tar?
You just cat the exe with the zip file, then it is all loaded into memory at the same time on process init. This is how e.g. LÖVE does game code packaging. (It can't be tar, because this trick only works because the PKZIP descriptor is at the end of the file.)
You can always use virtualized Linux to avoid the NTFS penalty (WSL2, VS Code dev containers, etc.)
Moving your whole workflow into WSL or nested containers just to dodge NTFS is a band-aid. Then you get flaky file watchers, odd perms, and a dev setup that feels like a workaround piled on top of another workaround. A fast Node VFS would remove a lot of this nonsense.
Oh it's a workaround for sure, didn't mean to suggest otherwise.
It’s insane to me that node works how it does. Zip files make so much more sense, I really liked that about Yarn.
Would it work to run a bundler over your code, so all (static) imports are inlined and tree shaken?