Comment by emilburzo
7 hours ago
I can empathize with them, this is actually quite difficult when you're an open source project.
You have your source code public, so you can't commit/push because then you've leaked info.
Your build pipeline is also likely public, so you can't just build from another source code... source.
You can create a private repo, but now you also need to build a private build system.
You don't get the same generous build resources for private repos, so you either:
a) simplify the build to just the essentials
b) pay
Unless you have significant funding, you'll probably go for a).
Now you have to maintain two systems, also with some differences between them.
Plus you need a way to get the build artifacts from the private system to the public one, typically at a fixed point in time.
Any failure/delay (e.g. less powerful build on private repo, errors because you exercise it rarely) causes people to get upset that you didn't respect the announced time.
Maybe you're also using things like image attestation, another can of worms.
Those are all challenges that surely need to be overcome!
But they had _two months_ to figure out how to overcome them. For them to not have figured out a solution and have pre-built images ready to push as soon as the embargo lifted is ridiculous.
Did they have the money and the people? PHP is chronically underfunded.
Linux Kernel approach of "just publish fix without fanfare" seems to be least compromising way to do it. Every bug can be security bug
A private build followed by a manual push would take literally two commands and could be done from a maintainer’s laptop… it is not rocket science.
I don't know about PHP, but at work our artifacts are signed, and you certainly cannot have the signing certificate on your laptop. It's stored in some secure server, and only accessible from some allowlisted build nodes, for specific builds (release branch, ...).
Now of course, there are ways around that if required, but they are, by design, manual, cumbersome and slow. I could not create a release build locally in a few minutes. A few hours maybe if the whole organisation dedicated itself to it.
Setting up all that for single man project might be a hassle but for any sofware with actual dev team I wouldn't call it hard to do.
I'd worry more about the fact that essentially publishing binary-first means that it ever gets compromised that makes it easier for attacker to smuggle exploit using "this is security update" pretext.
I'd say that for someone like PHP, in this era of AI-driven vulnerability/patch race, having a private CI/CD pipeline is a necessity.
PHP is written in C, I'd be surprised if similar problems didn't lurk in other places.
Maybe they will construct the private pipeline after this crisis. Millions of websites are probably now waiting for all the downstream Docker images to be built and pushed.
The images everyone is waiting for are built by Docker though? So now if you're Docker you have to have a private CI/CD pipeline with all library projects?
It would be great, but it's also something to manage. Just keeping it working with all the changes that happen across all projects would be something.
All library projects, certainly not.
But the top 1 percent by usage/downloads, I can imagine, and PHP probably qualifies into that top slice.
Making a cloud CI/build/release system whose behavior can't be reproduced locally requires some kind of brain damage. I can't believe people tolerate this.
That's... not what I said at all?
But let's go with that scenario, now you have all sorts of problems, from practical to security, like:
- accidentally/maliciously building the wrong thing? there's a patch, people agreed to it, but the wrong commit is built
- building for multiple architectures/versions?
- signing keys on someone's laptop?
- worse, image push/write keys on someone's laptop?
And so on.
The problems you describe in your first post are only problems when you can't reproduce your build process locally. With a local build you can always do it in private for free.
The problems you bring up here are eminently solvable too. Yes there are many solutions for building for multiple architectures and versions. Cross builds, containers, VMs, or just having more than one computer locally. You can address building the wrong thing with reproducible builds, but for a one off patch like this it's not the most likely problem and relying on a trusted maintainer temporarily during an emergency is better than waiting for a glacially slow cloud build while exploits spread. Pushing builds to the cloud is inherently a cloud step, so it's fine to leave keys for that in the cloud, but there should be a process for publishing local builds and it should be essentially instant when the build is already done.
You should always have a local build process that replicates CI. There is no excuse.