Comment by woodruffw
9 hours ago
I’m not sure what this has to do with TFA: Python does have hash-pinning. TFA is not about modifying existing files on the index (PyPI doesn’t allow that), but about adding new files to a pre-existing release. But that doesn’t change the hash of older distributions on that release.
I feel you’re quibbling over semantics here.
In concept why can’t the full set of files in a release be a single, one-way hash value, with both adding or releasing changing the hash value?
Because uploads to PyPI are not atomic. They’re now capped within a 14 day window, but it would be extremely confusing to users to have the “release” hash of their dependencies change repeatedly.
(Also: we’d need to determine what it even means to hash a set of files. Do we order by canonicalized filename, by upload time, etc. Each of these has surprising implications!)
Edit: to be clear, it’s not ideal or fully intentional that uploads to PyPI are not atomic. But it’s the status quo and any change will take years.
These things you’re posing as design questions are already solved 1 million times over. If you don’t want to design something new, just use Git semantics. These are not excuses for inaction. Python package management has been a horrifying embarrassment for 15 years and there’s no excuse for the amount it’s been neglected as every other ecosystem has shown better ways.
If it weren’t for AI, Python would have died as a language ecosystem due to their perpetual neglect of this area. The python community clearly had no ability to solve this problem themselves. I got sick and left and started using better tools.
It’s only now that they’re the cornerstone of the world economy that other people are coming in and cleaning up their messes for them. It’s pitiful. Python failed and needed to be saved from itself.
The semantics are important. A release is composed of multiple distributions (sets of files). Each distribution does have a one-way hash value. This is what you lock to. Adding files does change the value of that hash. Files cannot be removed or changed.
You lock to the distribution instead of the release so your build doesn't have to download distributions for platforms you're not using.
Which is the entire problem. "Release" should be a signed hash of a manifest file that contains hashes of all the other files (in case of Git that's what commit does).
PyPi just decided to do what VCSes already did, worse
1. There are multiple levels at which things are hashed, signed and checked when it comes to Python packages. Eg. each file in the Wheel beside the RECORD file is hashed using SHA256. This hash is never checked :( You can also sign your packages (RECORD.jws anyone? Is that still supported?), but nobody checks that either.
2. There are hashes in the HTML served by PyPI. These are updated at the whim of both the index and the publisher. Even though they are checked by pip during install, they are worthless.
3. There are many ways to install packages that work around (2). Custom index server doesn't have to provide hashes, and pip will happily install that. You can install from sources, from a package you've downloaded somewhere, form VCS, you can build it during install, all without even prompting the user to confirm the very scary choices.
NB. I have no idea how do you make the leap from "adding files to release" to "not modifying the release". To me, adding file to release is sure as hell modifying it. Here's a very simple malicious example:
I release package "innocent" with an empty "scripts" section. Then, in the subsequent modification to this release, I add the "scripts" section with a script named "notebook". Now, whenever my user wants to run Jupyter notebook, they will call my "notebook" program, not the one from Jupyter package.