Comment by danpalmer
3 years ago
Debian is its own worst enemy with things like this. It’s why we eventually moved off it at a previous job, because deploying Python server applications on it was dreadful.
I’m sure it’s better if you’re deploying an appliance that you hand off and never touch again, but for evolving modern Python servers it’s not well suited.
Yes 1000x What is it with them which makes them feel entitled to have special "dist-packages" vs "site-packages" as is the default? This drives me nuts, when I have a bunch of native packages I want to bundle in our in-house python deployment. CentOS and Ubuntu are vanilla, and only Debian (mind-boggingly) deviates from the well-trodden path.
I still haven't figured out how to beat this dragon. All suggestions welcome!
> What is it with them which makes them feel entitled to have special "dist-packages" vs "site-packages" as is the default? This drives me nuts, when I have a bunch of native packages I want to bundle in our in-house python deployment. CentOS and Ubuntu are vanilla, and only Debian (mind-boggingly) deviates from the well-trodden path.
Hi, I'm one of the people that look after this bit of Debian (and it's exactly the same in Ubuntu, FWIW).
It's like that to solve a problem (of course, everything has a reason). The idea is that Debian provides a Python that's deeply integrated into Debian packages. But if you want to build your own Python from source, you can. What you build will use site-packages, so it won't have any overlap with Debian's Python.
Unfortunately, while this approach was designed to be something all package-managed distributions could do, nobody else has adopted it, and consequently the code to make it work has never been pushed upstream. So, it's left as a Debian/Ubuntu oddity that confuses people. Sorry about that.
My recommendations are: 1. If you want more control over your Python than you get from Debian's package-managed python, build your own from source (or use a docker image that does that). 2. Deploy your apps with virtualenvs or system-level containers per app.
Dist packages is the right way to handle Python libs. You'd prefer to have the distro package manager clashing with Pip? Never knowing who installed what. Breaking things when updates are made.
I usually make a venv in ~/.venv and then activate it at the top of any python project. Makes it much easier to deal with dependencies when they're all in one place.
i am a big fan of .venv/ -- except when it takes ~45 mins to compile the native extension code in question -- then I want it all pre-packaged.
11 replies →
second this and it's what I do on all Linux distros, just run it inside .venv as the site-installation.
if you need extra dependencies that pip can not do well in the .venv case, Conda can help with its own and similar site-based installation.
I don't know how it is different in the python installation case between ubuntu and debian, they seem the same to me.
IMO bespoke containers using whatever python package manager makes sense for each project. Or make the leap to Nix(OS) and then still have to force every python project into compliance which can be very easy if the PyPy packages you need are already in the main Nix repo (nixpkgs) or very difficult if depends on a lot of uncommon packages, uses poetry, etc.
Since PEP 665 was rejected the Python ecosystem continues to lack a reasonable package manager and the lack of hashed based lock files prevents building on top of the current python project/package managers.
dist packages are a must for software written in Python that is part of the distribution itself.
You're not really answering why they are important?
Is it because .deb packages will install inside dist-packages and when you run pip install as root without a virtual env, it installs inside site-packages?
I don't really see how this helps though? Sure you won't get paths to clash between the two but you still have duplicate packages which is probably not what you want..
8 replies →
> I still haven't figured out how to beat this dragon. All suggestions welcome!
Docker
What distro did you move to? IME debian as a base image for python app containers is also kind of a pain.
We moved to stripped down Debian images in containers and made sure to not use any of the Debian packaging ecosystem.
It works completely fine in my experience.
Lucky you. Having gone through multiple Debian upgrades, a Python 2->3 migration on Debian, and Debian python packaging to Pip/PyPI, it was a whole world of pain that cost us months of development time over years, as well as a substantial amount of downtime.