Comment by lrvick
6 hours ago
The linux distribution I co-maintain uses mold as our bootstrap linker to bootstrap rust itself, and it saved us -hours- on long version-by-version build chains. Mold being in c meant we could build it very early and use it as the default linker distro wide and enjoy build speedups everywhere.
Now sadly we will have to fork and maintain the c version as mold2 forever.
Rust is not actually the right tool for all problems.
Like some other commenters, I do not understand why using a cached binary isn’t sufficient. Why not use the mold3 binary and build that first, and then cache it and never rebuild it again? I’m guessing it has to do with hermiticity and build provenance guarantees in your distro.
Do you have any reading material that you can share that might help me understand better? Docs for the distro, or an issue tracker I can search through?
https://stagex.tools
In short, the entire distro is always built in one-shot at any given commit, and we can only rely on cached binaries from a past release if they or nothing in their supply chains changed. Given rustc depends on almost everything, mold3 would be built far too late to be useful for the most expensive build in the whole tree, which is rust.
Recursive dependencies would break our threat model, so we cannot use any rust tools to bootstrap rust. We bootstrap rust from llvm which we bootstrap from gcc which we bootstrap from tinycc which we bootstrap from M2Planet and so on back to 186 bytes of machine code.
Anyone getting a rust package from stagex must be able to build the entire tree up until that package and get the same hash, with no binary dependencies, thus removing any trust in maintainers.
What _is_ your threat model?
If you can boostrap one version of stagex (so that its mold is trustworthy), I don't see why you can't use that to build another version of stagex.
As you say, you can rely on the cached binary because nothing in the old version's supply chain would change; it is pinned, and as long as you can build that, everything is fine.
1 reply →
This is entirely self-inflicted on your part!
> Rust is not actually the right tool for all problems.
Rust is certainly the right tool for this problem, your own decisions notwithstanding.
1 reply →
Why not use LLD for linking Rust? You already must have the LLVM for Rust to be buildable, so that do not add any dependency.
We use the llvm linker, lld, to link mold at the earliest point we can, then use mold exclusively after that. We are an LLVM native distro so we build llvm right away as the system compiler used to build the whole tree instead of gcc.
LLD would work fine for the whole tree, and did previously, but is much much slower than mold which is why we switched to mold.
Having to build all dependencies of rust including python and openssl and everything else before being able to use mold erases most of the full-tree build speedups as the path to rust is already about 80% of the full tree build time.
We are a distro that mandates independently verified 100% deterministic builds from source for any given release commit of the tree, so we have to build the whole tree several times for every release.
Can't you just keep using Mold2 for the initial toolchain bootstrap and then build mold3 with Rust as part of the "final" set of toolchains that are to be used to compile everything beyond that? It isn't perfect but it's congruent with how many other components in most open reproducible bootstrapping flows work; ie using GCC 4.4 or whatever just to bootstrap GCC 10. It's annoying but it only needs to be done once and then you just use it to bootstrap newer components forever. Doesn't help with compile times, though.
2 replies →
Agreed -- not my wheelhouse, but couldn't you just build rust earlier in your process using LLD, then build mold, then build everything else?
Would that meant that building rust is slower, but everything else is the same, and you don't have to maintain a fork of another complex project?
Rust requires python and perl and musl and openssl and 80% of the entire wall time of building the whole tree.
Rust is the single most expensive thing to bootstrap in any given linux distro.
It is the thing you need mold the most for to speed things up.
2 replies →
Because mold is faster. Linking is a significant bottleneck when building large projects.
But we're just talking for bootstrapping Rust here.
The complaint is about bootstrapping, where speed is of little relevance.
2 replies →
I have been thinking about Rust bootstrapping recently. Couldn't a Rust compiler without borrow checker be put together relatively easily?
Assuming the source code contains no issues (which can be checked later once the Rust compiler is built), one could leave that piece behind, and take the shortest path from .rs to executed code (C transpilation, or even an interpreter).
This exists: https://github.com/thepowersgang/mrustc.
It seems like you would benefit from having reproducible and hermetic builds with a good caching layer so you don't do the same work again and again.
Then you can just use the latest built version to build mold and the next version of rust.
Our tree is entirely reproducible and hermetic. In fact we are the only distro that does this 100%.
The problem is changing any dependency of rust, even python or perl or musl or openssl or the llvm stack or any dependencies of the llvm stack have the potential of resulting in a different hash for rust.
Any time a dependency is changed all decedents must be rebuilt. Which we must do very frequently. That is where mold saved us a ton of time.
That's why you build a stage2 compiler at least with stage0 being the latest prebuilt.
Your stage1 will be based on whatever version is your stage0, but your stage2 should end up identical to any other build with any other stage0. So you don't have to rebuild the full chain all the time.
And then preferably you build a stage3 with FDO, you can easily get 20% more speed with a good corpus of tests.
Do you have at least a distributed cache with I guess sccache to save time for each rebuild?
1 reply →
If the need is well justified, maybe there is great chance to ask adding #[no_mangle] and extern C support? Since release is very fresh. If that is causing the problem. Or is some dependency the issue?
The need here is bootstrapping. If mold is written in Rust you cannot even compile it.
Even Rust is written with Rust. But yeah, if you want to bootstrap absolutely from the beginning, I see the issue.
2 replies →
Could you not either bootstrap rust with lld or bootstrap rust with mold2?
Bootstrapping rust with mold2 is going to be the likely play. My complaint is that now we must maintain mold2 forever now as mold3 rust edition is now incompatible with most of our tree which is made up of dependencies of rust.
How much work is involved with mantaining a linker in deep maintenance mode?
1 reply →
Have you considered using Eurydice?
I am not finding any linkers by that name.
It's a rust to C transpiler, I was thinking it could let you use a C version of the mold linker without maintaining one yourself.
1 reply →
how's bootstrapping rust story nowadays?
You could use https://github.com/thepowersgang/mrustc to get relatively modern Rust compiler (1.90 currently apparently, but every now and then that is updated). Then build newer rustc from there to reach the current 1.99.
But I don't get why some people are obsessed with bootstrapping. Yes it is good to be able to do it, but it isn't something you need to do regularly.
Especially since rust had a much better cross compilation story than C or C++ (not as good as go or zig though), so you don't need to bootstrap on a new architecture, just cross compile to it. Furthermore, new architectures for hosting a compiler (as opposed to just a target, like microcontrollers) is a rare event. Just something that happens every few years.
> But I don't get why some people are obsessed with bootstrapping.
It only matters if you have supply chain attacks in your threat model. Given they are up 400x since 2019, they should probably be in almost every threat model. Most distros operate on the honor system and that is not going to survive the post AI world.
2 replies →
I'm a bit confused why a decision to decrease the maintenance burden of Mold by switching languages makes Rust the wrong tool here? Fearless concurrency sounds like a huge benefit for what they're doing, given the resources they have.
It's simply an ordering problem. When building the entire world from scratch, usually the C and C++ toolchains are built near the very first, and Rust toolchains built somewhat later. Anything written in Rust must come after the Rust toolchain is built. You need a linker as part of your C++ toolchain, so it must be written in a language ready to go at that point. If it is written in C, you are done. If it is written in Rust you have to wait. So a Rust-based linker can no longer be used from that very, very early C bootstrap.
It's not a big deal for normal users, where you have Rust ready to go. Kind of a bummer in this case, but this is a specialized one.
Can't you just use a statically linked mold binary for the initial bootstrap, just like you use some kind of pre-existing, basic C compiler?
4 replies →
Maybe that's about to flip. Maybe Rust will be built first, followed by C.
6 replies →
One of the explicit goals for Mold 3.0 is to promote its use as the default linker in Linux distributions, and bootstrapping complexity is certainly worthy of consideration in that realm.
> We will then conduct extensive compatibility testing and work closely with Linux distribution developers to make it practical for them to adopt mold as /usr/bin/ld. Making this happen is one of our highest priorities for mold 3.x.
We are one of only two distros that uses mold as the default system linker. Unfortunately that will be stuck at mold2 until the full tree is built, but we can swap to mold3 for end user consumers of the tree. Sucks that we have to maintain and use both now.
[flagged]
I do understand what you are trying to say, but I think that this fails to be kind to Irvick and the other maintainers of the stage0 project who are actually being quite underfunded (someone like OAI should fund them in the name of security!)
So asking them to pay money is well.. not quite the solution.
What can happen as it often happens, is this, Mold creator created the project, Stage0 found it useful, Mold ports itself to rust, Stage0 now founds it not useful, Stage0 can comment on the new update and be slightly disappointed and comment how they would've preferred to not rust in this particular case for them.
Yes this doesn't prevent mold from changing to rust or anything as it was shown but I guess we can allow the ability for Irvick to drop a comment I guess without saying that's your responsibility while they are just being maintainers of the open source and an underfunded/mostly volunteer one of work at that.
Also, @Irvick, stage0 is extremely cool, I hope that a lot more companies sponsor the work that stage0 contributors are doing. I think that it can help the supply-chain issues that the industry is facing at, and is honestly just a really cool idea and I love reading your comments and thank you!
To be clear we maintain stagex, which is a distro built on stage0 as the foundation.
Otherwise thank you for for the kind words.
1 reply →
>Rust is not actually the right tool for all problems.
"my problems (that are not mold's) are not solved by mold. How dare mold make those decisions?"
Maybe you should rewrite more of your linux distribution in rust so it's available earlier in the build process and get back to it being the default linker.
They were solved by mold. And then mold decided to un-solve them.
The mold developers are not responsible for what people want to do with their software.
No, they were never one of mold's goals. You don't get to assign solutions to people _and then blame them_. Hyrum's Law being a load bearing part of your infrastructure isn't Mold's problem.
10 replies →
Luckily Mr Clanker can do most of the work for you at least (I myself have already had Claude/Codex rewrite a couple of Rust projects in C, worked great)
And repeat that work every time you sync with upstream?
Why do you need to bootstrap a toolchain to bootstrap a distro?
I know this is common but it seems like either an aesthetic decision, or glibc cruft.
There are classes of virus that are hard to detect. One is a compiler virus that passes itself from compiler to compiler. You only get rid of the vector by bootstrapping from 0.
No, you can do bootstrapping and save binaries for reuse with hash verification. Android did that for its Rust toolchain: https://cs.android.com/android/platform/superproject/main/+/...
Bootstrapping at every build does not save you from the threat you think it does.
1 reply →
Sure but that's a compiler bootstrapping problem. It doesn't answer the question: why do you need to bootstrap the toolchain to build the distro? You can reuse a trusted toolchain that's been safely bootstrapped .
1 reply →