← Back to context

Comment by zahlman

2 days ago

> Serious question: is that solved?

It depends on what "setting up" means.

Creating an environment, given that the Python binary it will use is already installed, is trivial (standard library functionality since late 2012). So is choosing which environment to use. So is installing pre-built packages, and even legacy source packages are pretty easy (but slow, and installation runs arbitrary code which is entirely needless for these) when they only contain pure Python code. Even dependency resolution is usually not too bad.

The big problems are things like

* building multi-language packages from source locally, because this is expected to set up temporary local build environments (and build tools have to avoid recursion there)

* using external non-Python dependencies (essentially unsolved, and everyone works around this by either vendoring stuff or by not declaring the dependency and failing at runtime) — see https://pypackaging-native.github.io/ for an overview of the problems and https://peps.python.org/pep-0725/ for what they're trying to standardize to deal with it

* dealing with metadata for source packages; in the really general case you have to build the source to get this (although the package-building API now provides a hook so that build backends can specifically prepare metadata). This is mainly because some packages have dependencies that depend on very particular platform details that (apparently) can't be expressed with the "environment marker" scheme in standard metadata (https://peps.python.org/pep-0508/#environment-markers)

* and, of course, figuring out which packages need to be in your environment (Python won't decide for you what your direct dependencies are) and managing that environment over time. The reason all these other tools popped up is because Pip only installs the packages and offers very basic environment inspection; it's only now starting to do anything with lockfiles, for example, now that there is finally a standard for them (https://peps.python.org/pep-0751/).

But if you mean, is there a standard toolchain that does everything and will be officially blessed by the core language developers, then no, you should not ever expect this. There is no agreement on what "everything" entails, and Python users (a large fraction of which don't fit the traditional image of a "developer" at all) have widely varying workflows and philosophical/aesthetic preferences about that. Besides which, the core language team doesn't generally work on or care about the problem; they care about the interpreter first and foremost. Packaging is an arms-length consideration. Good news, though: the Python Packaging Authority (not at all authoritative, and named with tongue firmly in cheek, but a lot of people didn't get that) is stepping up and working on official governance (see https://peps.python.org/pep-0772/).

> at last count my laptop has 38 python binaries

Something has gone very wrong (unless you're on Windows, such that admin rights would be needed to create symlinks and by default `venv` doesn't try). To be clear, I mean with your setup, not with the tooling. You should only need one per distinct version of Python that your various environments use. I'd be happy to try to help if you'd like to shoot me an email (I use that Proton service, with the same username as here) and give more details on how things are currently set up and what you're trying to accomplish that way.