Comment by tetha

1 day ago

And strictly speaking you cannot trust the pip upgrade of pip with the vulnerable cert in the trust store due to certifi, which you also cannot upgrade safely with pip for the same reason. All of these are at one out of two compromises from being MITM'd during the upgrade.

So strictly we have to burn everything python to the ground and reinstall entirely.

9 comments

tetha

saltcured 20 hours ago

Heh. This is part of why I always preferred getting my python runtime and all these core libs from my linux distro packager. But python die-hards look at me like I'm insane when I suggest this.

Even in today's LLM-addled world, I still think it's a better tradeoff for smaller system/site operators. Get the distro-level testing and regular auto-patching, rather than telling yourself it is a good idea to "freeze" your python world at the pace of your slowest app/system integrator.

It's the same reason I think docker is an anti-pattern for these small groups. You really cannot safely use these strategies to bake an image and divorce yourself from the maintained distro layer unless you are going to adopt CI/CD that does the same thing. Keep tracking all these patches to the entire runtime with all the dependencies, rebuild, re-test, and redeploy.

It's total cargo culting to adopt the freeze/image techniques and just abandon apps in place as if that is robust!

But, it's also more than just automatically updating everything. There is an editorial aspect to distro management to help secure the supply chain to your app. Most smaller orgs cannot feasibly do this as well as the regular linux distro maintainers. Some large orgs can probably do it better, but that doesn't matter if you aren't doing it large org style with dedicated security folks tracking all this against well-developed policies and procedures...

zahlman 7 hours ago

> But python die-hards look at me like I'm insane when I suggest this.
They have a point. The distro is only carrying a small fraction of what's available on PyPI, and will have out-of-date versions that may lack security patches (and for projects like yt-dlp, are highly unlikely to work at all). Realistically they aren't applying security backports; there would rarely be a reason not to just use the new version, and the resources just aren't there for that scale of patching anyway.
Further, pip will interfere with your system package manager's tracking of the system environment (PEP 668), and any attempt you make to install from .deb/.rpm/etc. to a virtual environment is a hack at best. Maybe you could symlink individual package folders to the system environment while installing a superset, but ugh. Oh, and if you're a developer (like most "Python die-hards" are) then this isn't very helpful, as you aren't testing the installation of your package the way that your users will do it.
graemep 17 hours ago

I agree about getting libs (binaries especially) from my distro, and it generally seems to be regarded as the "wrong thing to do". Freezing anything that might mean you get security updates slower seems to be a mistake to me.

aftbit 20 hours ago

Or - you can just upgrade pip with a SHA hash pin, or set the magic env vars to use the system bundle.

You do need to be careful about creating new venvs with the old version of pip bundled in ensurepip in older Python releases. `uv` kindly patches that to newest pip, but not every Python binary does.

  python -m pip install "pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70"

Or alternatively, on Debian with fixed system cert chain:

  PIP_USE_DEPRECATED=legacy-certs PIP_CERT=/etc/ssl/certs/ca-certificates.crt python -m pip install --upgrade "pip>=26.2.2"

https://discuss.python.org/t/avoiding-compromised-certificat...

zahlman 7 hours ago

I had a pretty fun time with it myself. Rather than a system package for pip, or the `ensurepip`-provided one, I've been using the copy managed by pipx. I had to:
* Update system certificates
* Locate that pip and directly invoke it to self-update with hash pinning, rather than going through pipx
* Fix a few stray venvs that contained their own pip
* Use the fixed pipx to update uv (although I guess the uv self-update should work even for a pipx-installed copy?)
The legacy-certs hint is neat. I did it the first way but would rather not have since I'm on Mint.

tetha 18 hours ago

Good to know. Does the first invocation recompute and check the hash of the wheel locally afterwards to ensure integrity?

aftbit 17 hours ago

Yep! I tested it with my own vibe-coded MITM proxy that hijacks pypi installations, which I created for ~$0.30 spend on DeepSeek API.

    $ python -m pip --no-cache install "pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70"
    Collecting pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70
      Downloading pip-26.2.2-py3-none-any.whl (1.8 MB)
         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 570.0 MB/s  0:00:00
    ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
        pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70 from https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70:
            Expected sha256 fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70
                 Got        8c448177668e735cdde7d6178694b90c084ba18559348cc394c4b1546eeec89c

The second one works too:

    $ PIP_USE_DEPRECATED=legacy-certs PIP_CERT=/etc/ssl/certs/ca-certificates.crt python -m pip install --no-cache --upgrade "pip>=26.2.2"
    WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082)'))': /simple/pip/
    WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082)'))': /simple/pip/
    WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082)'))': /simple/pip/
    WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082)'))': /simple/pip/
    WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082)'))': /simple/pip/
    Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082) - skipping
    ERROR: Could not find a version that satisfies the requirement pip>=26.2.2 (from versions: none)
    ERROR: No matching distribution found for pip>=26.2.2

As compared with a naive installation through the proxy:

    $ python -m pip --no-cache install 'pip>=26.2.2'
    Collecting pip>=26.2.2
      Downloading pip-26.2.2-py3-none-any.whl.metadata (4.6 kB)
    Downloading pip-26.2.2-py3-none-any.whl (1.8 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 775.3 MB/s  0:00:00
    Installing collected packages: pip
      Attempting uninstall: pip
        Found existing installation: pip 26.2.1
        Uninstalling pip-26.2.1:
          Successfully uninstalled pip-26.2.1
    Successfully installed pip-26.2.2

    $ cat venv/lib/python3.14/site-packages/sni_proxy_sentinel.txt
    SNI proxy lab sentinel. This file is inert.

1 reply →

zahlman 8 hours ago

> All of these are at one out of two compromises from being MITM'd during the upgrade.

This is not true; pip supports hash pinning, so an attacker would have to be able to defeat SHA256 as well (or compromise both the official Python discussion forum and PyPI itself, so as to forged the published hash). Other hashes can also be used. Instructions are provided on the Python discussion forum to self-upgrade pip with hash pinning: https://discuss.python.org/t/information-for-python-users-re...