The other issue is that people seem to just copy configure/autotools scripts over from older or other projects because either they are lazy or don't understand them enough to do it themselves. The result is that even with relatively modern code bases that only target something like x86, arm and maybe mips and only gcc/clang, you still get checks for the size of an int, or which header is needed for printf, or whether long long exists.... And then the entire code base never checks the generated macros in a single place, uses int64_t and never checks for stint.h in the configure script...
I don't think it's fair to say "because they are lazy or don't understand". Who would want to understand that mess? It isn't a virtue.
A fairer criticism would be that they have no sense to use a more sane build system. CMake is a mess but even that is faaaaar saner than autotools, and probably more popular at this point.
Simple projects: just use plain C. This is dwm, the window manager that spawned a thousand forks. No ./configure in sight: <https://git.suckless.org/dwm/files.html>
If you run into platform-specific stuff, just write a ./configure in simple and plain shell: <https://git.suckless.org/utmp/file/configure.html>. Even if you keep adding more stuff, it shouldn't take more than 100ms.
If you're doing something really complex (like say, writing a compiler), take the approach from Plan 9 / Go. Make a conditionally included header file that takes care of platform differences for you. Check the $GOARCH/u.h files here:
It’s always wise to be specific about the sizes you want for your variables. You don’t want your ancient 64-bit code to act differently on your grandkids 128-bit laptops. Unless, of course, you want to let the compiler decide whether to leverage higher precision types that become available after you retire.
I did something like the system described in this article a few years back. [1]
Instead of splitting the "configure" and "make" steps though, I chose to instead fold much of the "configure" step into the "make".
To clarify, this article describes a system where `./configure` runs a bunch of compilations in parallel, then `make` does stuff depending on those compilations.
If one is willing to restrict what the configure can detect/do to writing to header files (rather than affecting variables examined/used in a Makefile), then instead one can have `./configure` generate a `Makefile` (or in my case, a ninja file), and then have the "run the compiler to see what defines to set" and "run compiler to build the executable" can be run in a single `make` or `ninja` invocation.
The simple way here results in _almost_ the same behavior: all the "configure"-like stuff running and then all the "build" stuff running. But if one is a bit more careful/clever and doesn't depend on the entire "config.h" for every "<real source>.c" compilation, then one can start to interleave the work perceived as "configuration" with that seen as "build". (I did not get that fancy)
You should use double underscores on attribute names to avoid conflicts with macros (user-defined macros beginning with double underscores are forbidden, as identifiers beginning with double underscores are reserved).
#if __has_attribute(__cold__)
# warning "This works too"
#endif
static void __attribute__((__cold__))
foo(void)
{
// This works too
}
The problem is that the various `__has_foo` aren't actually reliable in practice - they don't tell you if the attribute, builtin, include, etc. actually works the way it's supposed to without bugs, or if it includes a particular feature (accepts a new optional argument, or allows new values for an existing argument, etc.).
yep. C's really come a long way with the special operators for checking if attributes exist, if builtins exist, if headers exist, etc.
Covers a very large part of what is needed, making fewer and fewer things need to end up in configure scripts. I think most of what's left is checking for items (types, functions) existence and their shape, as you were doing :). I can dream about getting a nice special operator to check for fields/functions, would let us remove even more from configure time, but I suspect we won't because that requires type resolution and none of the existing special operators do that.
Noticed an easter egg in this article. The text below "I'm sorry, but in the year 2025, this is ridiculous:" is animated entirely without Javascript or .gif files. It's pure CSS.
It's a bit of a balance once you get bigger dependencies. A generic autoconf is annoying to write, but rarely an issue when packaging for a distro. Most issues I've had to fix in nixpkgs were for custom builds unfortunately.
But if you don't plan to distribute things widely (or have no deps).. Whatever, just do what works for you.
Write your own configure? For an internal project, where much is under domain control, sure. But for the 1000s of projects trying to multi-plarform and/or support flavours/versions - oh gosh.
autoconf is in no way, shape or form an "official" build system associated with C. It is a GNU creation and certainly popular, but not to a "monopoly" degree, and it's share is declining. (plain make & meson & cmake being popular alternatives)
And on macOS, the notarization checks for all the conftest binaries generated by configure add even more latency. Apple reneged on their former promise to give an opt-out for this.
On the topic* of having 24 cores and wanting to put them to work: when I were a lad the promise was that pure functional programming would trivially allow for parallel execution of functions. Has this future ever materialized in a modern language / runtime?
x = 2 + 2
y = 2 * 2
z = f(x, y)
print(z)
…where x and y evaluate in parallel without me having to do anything. Clojure, perhaps?
*And superficially off the topic of this thread, but possibly not.
Superscalar processors (which include all mainstream ones these days) do this within a single core, provided there are no data dependencies between the assignment statements. They have multiple arithmetic logic units, and they can start a second operation while the first is executing.
But yeah, I agree that we were promised a lot more automatic multithreading than we got. History has proven that we should be wary of any promises that depend on a Sufficiently Smart Compiler.
Eh, in this case not splitting them up to compute them in parallel is the smartest thing to do. Locking overhead alone is going to dwarf every other cost involved in that computation.
That looks more like a SIMD problem than a multi-core problem
You want bigger units of work for multiple cores, otherwise the coordination overhead will outweigh the work the application is doing
I think the Erlang runtime is probably the best use of functional programming and multiple cores. Since Erlang processes are shared nothing, I think they will scale to 64 or 128 cores just fine
Whereas the GC will be a bottleneck in most languages with shared memory ... you will stop scaling before using all your cores
But I don't think Erlang is as fine-grained as your example ...
AFAIU Erlang is not that fast an interpreter; I thought the Pony Language was doing something similar (shared nothing?) with compiled code, but I haven't heard about it in awhile
There's some sharing used to avoid heavy copies, though GC runs at the process level. The implementation is tilted towards copying between isolated heaps over sharing, but it's also had performance work done over the years. (In fact, if I really want to cause a global GC pause bottleneck in Erlang, I can abuse persistent_term to do this.)
Bend[1] and Vine[1] are two experimental programming languages that take similar approaches to automatically parallelizing programs; interaction nets[3]. IIUC, they basically turn the whole program into one big dependency graph, then the runtime figures out what can run in parallel and distributes the work to however many threads you can throw at it. It's also my understanding that they are currently both quite slow, which makes sense as the focus has been on making `write embarrassingly parallelizable program -> get highly parallelized execution` work at all until recently. Time will tell if they can manage enough optimizations that the approach enables you to get reasonably performing parallel functional programs 'for free'.
> …where x and y evaluate in parallel without me having to do anything.
I understand that yours is a very simple example, but a) such things are already parallelized even on a single thread thanks to all the internal CPU parallelism, b) one should always be mindful of Amdahl's law, c) truly parallel solutions to various problems tend to be structurally different from serial ones in unpredictable ways, so there's no single transformation, not even a single family of transformations.
There have been experimental parallel graph reduction machines. Excel has a parallel evaluator these days.
Oddly enough, functional programming seems to be a poor fit for this because the fanout tends to be fairly low: individual operations have few inputs, and single-linked lists and trees are more common than arrays.
I believe it's not the language preventing it but the nature of parallel computing. The overhead of splitting up things and then reuniting them again is high enough to make trivial cases not worth it. OTOH we now have pretty good compiler autovectorization which does a lot of parallel magic if you set things right. But it's not handled at the language level either.
there have been fortran compilers which have done auto parallelization for decades, i think nvidia released a compiler that will take your code and do its best to run it on a gpu
this works best for scientific computing things that run through very big loops where there is very little interaction between iterations
As a user I highly appreciate ./configure for the --help flag, which usually tells me how to build a program with or without particular functionalities which may or may not be applicable to my use-case.
I get the impression configure not only runs sequentially, but incrementally, where previous results can change the results of tests run later. Were it just sequential, running multiple tests as separate processes would be relatively simple.
Also, you shouldn’t need to run ./configure every time you run make.
No, but if you are doing something like rebuilding a distro's worth of packages from source from scratch, the configure step starts to dominate. I build around 550, and it takes around 6 hours on a single node.
Most checks are common, so what can help is having a shared cache for all configure scripts so if you have 400 packages to rebuild, it doesn't check 400 times if you should use flock or fcntl. This approach is described here:
https://jmmv.dev/2022/06/autoconf-caching.html
It doesn't help that autoconf is basically abandonware, with one forlorn maintainer trying to resuscitate it, but creating major regressions with new releases:
https://lwn.net/Articles/834682/
Very nice! I always get annoyed when my fancy 16 thread CPU is left barely used as one thread is burning away with the rest sitting and waiting. Bookmarking this for later to play around with whatever projects I use that still use configure.
Also, I was surprised when the animated text at the top of the article wasn't a gif, but actual text. So cool!
Agreed! The CMake Xcode generator is extremely slow because not only is it running the configure tests sequentially, but it generates a new Xcode project for each of them.
Why do we need to even run most of the things in ./configure? Why not just have a file in /etc which is updated when you install various packages which ./configure can read to learn various stats about the environment? Obviously it will still allow setting various things with parameters and create a Makefile, but much faster.
Keep in mind that the build intentionally depends on environment variables, people often install non-packaged dependencies in bad ways, and cross-compiling is a thing, so it's not that simple.
(The conclusion I distilled out of reading that at the time, I think, was that this is actually sort of happening, but slowly, and autoconf is likely to stick around for a while, if only as a compatibility layer during the transition.)
It is possible in theory to speed up existing configure scripts by switching interpreter from /bin/sh to something that scans file, splits it to independent blocks and runs them in parallel.
Historically, different Unixes varied a lot more than they do today. Say you want your program to use the C library function foo on platforms where it’s available and the function bar where it isn’t: You can write both versions and choose between them based on a C preprocessor macro, and the program will use the best option available for the platform where it was compiled.
But now the user has to set the preprocessor macro appropriately when he builds your program. Nobody wants to give the user a pop quiz on the intricacies of his C library every time he goes to install new software. So instead the developer writes a shell script that tries to compile a trivial program that uses function foo. If the script succeeds, it defines the preprocessor macro FOO_AVAILABLE, and the program will use foo; if it fails, it doesn’t define that macro, and the program will fall back to bar.
That shell script grew into configure. A configure script for an old and widely ported piece of software can check for a lot of platform features.
I'm not saying we should send everyone a docker container with a full copy of ubuntu, electron and foo.js whether they have foo in their c library or not, but maybe there is a middle ground?
Hands have to get dirty somewhere. "As deep as The Worker's City lay underground, so high above towered the City of Metropolis."
The choices are:
1. Restrict the freedom of CPU designers to some approximation of the PDP11. No funky DSP chips. No crazy vector processors.
2. Restrict the freedom of OS designers to some approximation of Unix. No bespoke realtime OSes. No research OSes.
3. Insist programmers use a new programming language for these chips and OSes. (This was the case prior to C and Unix.)
4. Insist programmers write in assembly and/or machine code. Perhaps a macro-assembler is acceptable here, but this is inching toward C.
The cost of this flexibility is gross tooling to make it manageable. Can it be done without years and years of accrued M4 and sh? Perhaps, but that's just CMake and CMake is nowhere near as capable as Autotools & friends are when working with legacy platforms.
> is this really a big deal given you run ./configure once
I end up running it dozens of times when changing versions, checking out different branches, chasing dependencies.
It’s a big deal.
> it's like systemd trading off non-determinism for boot speed, when it takes 5 minutes to get through the POST
5 minute POST time is a bad analogy. systemd is used in many places, from desktops (that POST quickly) to embedded systems where boot time is critical.
If deterministic boot is important then you would specify it explicitly. Relying on emergent behavior for consistent boot order is bad design.
The number of systems that have 5 minute POST times and need deterministic boot is an edge case of an edge case.
This aspect of configure, in particular, drives me nuts. Obviously I'd like it to be faster, but it's not the end of the world. I forget what I was trying to build the other week, but I had to make 18 separate runs of configure to find all the things I was missing. When I dug into things it looked like it could probably have done it in 2 runs, each presenting a batch of things that were missing. Instead I got stuck with "configure, install missing package" over and over again.
> I end up running it dozens of times when changing versions, checking out different branches, chasing dependencies.
Yeah... but neither of that is going to change stuff like the size of a data type, the endianness of the architecture you're running on, or the features / build configuration of some library the project depends on.
Parallelization is a bandaid (although a sorely needed!) IMHO, C/C++ libraries desperately need to develop some sort of standard that doesn't require a full gcc build for each tiny test. I'd envision something like nodejs's package.json, just with more specific information about the build details themselves. And for the stuff like datatype sizes, that should be provided by gcc/llvm in a fast-parseable way so that autotools can pick it up.
If you do a lot of bisecting, or bootstrapping, or building compatibility matrices, or really anything that needs you to compile lots of old versions, the repeated ./configure steps really start feeling like a drag.
In a "reasonably well-behaved program", if you have the artifacts from a current configure, like a "config.h" header, they are compatible with older commits, even if configurations changed, as long as the configuration changes were additive: introducing some new test, along with a new symbol in "config.h".
It's possible to skip some of the ./configure steps. Especially for someone who knows the program very well.
> it's like systemd trading off non-determinism for boot speed, when it takes 5 minutes to get through the POST
That's a bad analogy: if a given deterministic service ordering is needed for a service to correctly start (say because it doesn't start with the systemd unit), it means the non-deterministic systemd service units are not properly encoding the dependencies tree in the Before= and After=
When done properly, both solutions should work the same. However, the solution properly encoding the dependency graph (instead of just projecting it on a 1-dimensional sequence of numbers) will be more flexible: it's the better solution, because it will give you more speed but also more flexibility: you can see the branches any leaf depends on, remove leaves as needed, then cull the useless branches. You could add determinism if you want, but why bother?
It's like using the dependencies of linux packages, and leaving the job of resolving them to package managers (apt, pacman...): you can then remove the useless packages which are no longer required.
Compare that to doing a `make install` of everything to /usr/local in a specific order, as specified by a script: when done properly, both solutions will work, but one solution is clearly better than the other as it encodes more finely the existing dependencies instead of projecting them to a sequence.
You can add determinism if you want to follow a sequence (ex: `apt-get install make` before adding gcc, then add cuda...), or you can use meta package like build-essentials, but being restricted to a sequence gains you nothing.
The other issue is that people seem to just copy configure/autotools scripts over from older or other projects because either they are lazy or don't understand them enough to do it themselves. The result is that even with relatively modern code bases that only target something like x86, arm and maybe mips and only gcc/clang, you still get checks for the size of an int, or which header is needed for printf, or whether long long exists.... And then the entire code base never checks the generated macros in a single place, uses int64_t and never checks for stint.h in the configure script...
I don't think it's fair to say "because they are lazy or don't understand". Who would want to understand that mess? It isn't a virtue.
A fairer criticism would be that they have no sense to use a more sane build system. CMake is a mess but even that is faaaaar saner than autotools, and probably more popular at this point.
This.
Simple projects: just use plain C. This is dwm, the window manager that spawned a thousand forks. No ./configure in sight: <https://git.suckless.org/dwm/files.html>
If you run into platform-specific stuff, just write a ./configure in simple and plain shell: <https://git.suckless.org/utmp/file/configure.html>. Even if you keep adding more stuff, it shouldn't take more than 100ms.
If you're doing something really complex (like say, writing a compiler), take the approach from Plan 9 / Go. Make a conditionally included header file that takes care of platform differences for you. Check the $GOARCH/u.h files here:
<https://go.googlesource.com/go/+/refs/heads/release-branch.g...>
(There are also some simple OS-specific checks: <https://go.googlesource.com/go/+/refs/heads/release-branch.g...>)
This is the reference Go compiler; it can target any platform, from any host (modulo CGO); later versions are also self-hosting and reproducible.
> either they are lazy or don't understand them enough to do it themselves.
Meh, I used to keep printed copies of autotools manuals. I sympathize with all of these people and acknowledge they are likely the sane ones.
It’s always wise to be specific about the sizes you want for your variables. You don’t want your ancient 64-bit code to act differently on your grandkids 128-bit laptops. Unless, of course, you want to let the compiler decide whether to leverage higher precision types that become available after you retire.
I did something like the system described in this article a few years back. [1]
Instead of splitting the "configure" and "make" steps though, I chose to instead fold much of the "configure" step into the "make".
To clarify, this article describes a system where `./configure` runs a bunch of compilations in parallel, then `make` does stuff depending on those compilations.
If one is willing to restrict what the configure can detect/do to writing to header files (rather than affecting variables examined/used in a Makefile), then instead one can have `./configure` generate a `Makefile` (or in my case, a ninja file), and then have the "run the compiler to see what defines to set" and "run compiler to build the executable" can be run in a single `make` or `ninja` invocation.
The simple way here results in _almost_ the same behavior: all the "configure"-like stuff running and then all the "build" stuff running. But if one is a bit more careful/clever and doesn't depend on the entire "config.h" for every "<real source>.c" compilation, then one can start to interleave the work perceived as "configuration" with that seen as "build". (I did not get that fancy)
[1]: https://github.com/codyps/cninja/tree/master/config_h
Nice! I used to do something similar, don't remember exactly why I had to switch but the two step process did become necessary at some point.
Just from a quick peek at that repo, nowadays you can write
#if __has_attribute(cold)
and avoid the configure test entirely. Probably wasn't a thing 10 years ago though :)
You should use double underscores on attribute names to avoid conflicts with macros (user-defined macros beginning with double underscores are forbidden, as identifiers beginning with double underscores are reserved).
The problem is that the various `__has_foo` aren't actually reliable in practice - they don't tell you if the attribute, builtin, include, etc. actually works the way it's supposed to without bugs, or if it includes a particular feature (accepts a new optional argument, or allows new values for an existing argument, etc.).
yep. C's really come a long way with the special operators for checking if attributes exist, if builtins exist, if headers exist, etc.
Covers a very large part of what is needed, making fewer and fewer things need to end up in configure scripts. I think most of what's left is checking for items (types, functions) existence and their shape, as you were doing :). I can dream about getting a nice special operator to check for fields/functions, would let us remove even more from configure time, but I suspect we won't because that requires type resolution and none of the existing special operators do that.
1 reply →
GNU Parallel seems like another convenient approach.
It has no concept of dependencies between tasks, or doing a topological sort prior to running the task queue. GNU Make's parallel mode (-j) has that.
Noticed an easter egg in this article. The text below "I'm sorry, but in the year 2025, this is ridiculous:" is animated entirely without Javascript or .gif files. It's pure CSS.
This is how it was done: https://github.com/tavianator/tavianator.com/blob/cf0e4ef26d...
Unfortunately it forgets to HTML-escape the <wchar.h> etc.
Whoops! Forgot to do that when I switched from a ``` block to raw html
I've spent a fair amount of time over the past decades to make autotools work on my projects, and I've never felt like it was a good use of time.
It's likely that C will continue to be used by everyone for decades to come, but I know that I'll personally never start a new project in C again.
I'm still glad that there's some sort of push to make autotools suck less for legacy projects.
You can use make without configure. If needed, you can also write your own configure instead of using auto tools.
Creating a make file is about 10 lines and is the lowest friction for me to get programming of any environment. Familiarity is part of that.
It's a bit of a balance once you get bigger dependencies. A generic autoconf is annoying to write, but rarely an issue when packaging for a distro. Most issues I've had to fix in nixpkgs were for custom builds unfortunately.
But if you don't plan to distribute things widely (or have no deps).. Whatever, just do what works for you.
Write your own configure? For an internal project, where much is under domain control, sure. But for the 1000s of projects trying to multi-plarform and/or support flavours/versions - oh gosh.
3 replies →
To extend on sibling comments:
autoconf is in no way, shape or form an "official" build system associated with C. It is a GNU creation and certainly popular, but not to a "monopoly" degree, and it's share is declining. (plain make & meson & cmake being popular alternatives)
I've stopped using autotools for new projects. Just a Makefile, and the -j flag for concurrency.
cmake ftw
Or meson is a serious alternative to cmake (Even better than cmake imho)
CMake also does sequential configuration AFAIK. Is there any work to improve on that somewhere?
1 reply →
You mean cargo build
2 replies →
cmake uses configure, or configure-like too!
1 reply →
And on macOS, the notarization checks for all the conftest binaries generated by configure add even more latency. Apple reneged on their former promise to give an opt-out for this.
On the topic* of having 24 cores and wanting to put them to work: when I were a lad the promise was that pure functional programming would trivially allow for parallel execution of functions. Has this future ever materialized in a modern language / runtime?
…where x and y evaluate in parallel without me having to do anything. Clojure, perhaps?
*And superficially off the topic of this thread, but possibly not.
Superscalar processors (which include all mainstream ones these days) do this within a single core, provided there are no data dependencies between the assignment statements. They have multiple arithmetic logic units, and they can start a second operation while the first is executing.
But yeah, I agree that we were promised a lot more automatic multithreading than we got. History has proven that we should be wary of any promises that depend on a Sufficiently Smart Compiler.
Eh, in this case not splitting them up to compute them in parallel is the smartest thing to do. Locking overhead alone is going to dwarf every other cost involved in that computation.
5 replies →
That looks more like a SIMD problem than a multi-core problem
You want bigger units of work for multiple cores, otherwise the coordination overhead will outweigh the work the application is doing
I think the Erlang runtime is probably the best use of functional programming and multiple cores. Since Erlang processes are shared nothing, I think they will scale to 64 or 128 cores just fine
Whereas the GC will be a bottleneck in most languages with shared memory ... you will stop scaling before using all your cores
But I don't think Erlang is as fine-grained as your example ...
Some related threads:
https://news.ycombinator.com/item?id=31176264
AFAIU Erlang is not that fast an interpreter; I thought the Pony Language was doing something similar (shared nothing?) with compiled code, but I haven't heard about it in awhile
Yes, Erlang's zero-sharing model is what I think Rust should have gone for in its concurrency model. Sadly too few people have even heard of it.
There's some sharing used to avoid heavy copies, though GC runs at the process level. The implementation is tilted towards copying between isolated heaps over sharing, but it's also had performance work done over the years. (In fact, if I really want to cause a global GC pause bottleneck in Erlang, I can abuse persistent_term to do this.)
Bend[1] and Vine[1] are two experimental programming languages that take similar approaches to automatically parallelizing programs; interaction nets[3]. IIUC, they basically turn the whole program into one big dependency graph, then the runtime figures out what can run in parallel and distributes the work to however many threads you can throw at it. It's also my understanding that they are currently both quite slow, which makes sense as the focus has been on making `write embarrassingly parallelizable program -> get highly parallelized execution` work at all until recently. Time will tell if they can manage enough optimizations that the approach enables you to get reasonably performing parallel functional programs 'for free'.
[1] https://github.com/HigherOrderCO/Bend [2] https://github.com/VineLang/vine [3] https://en.wikipedia.org/wiki/Interaction_nets
> …where x and y evaluate in parallel without me having to do anything.
I understand that yours is a very simple example, but a) such things are already parallelized even on a single thread thanks to all the internal CPU parallelism, b) one should always be mindful of Amdahl's law, c) truly parallel solutions to various problems tend to be structurally different from serial ones in unpredictable ways, so there's no single transformation, not even a single family of transformations.
There have been experimental parallel graph reduction machines. Excel has a parallel evaluator these days.
Oddly enough, functional programming seems to be a poor fit for this because the fanout tends to be fairly low: individual operations have few inputs, and single-linked lists and trees are more common than arrays.
I believe it's not the language preventing it but the nature of parallel computing. The overhead of splitting up things and then reuniting them again is high enough to make trivial cases not worth it. OTOH we now have pretty good compiler autovectorization which does a lot of parallel magic if you set things right. But it's not handled at the language level either.
Jax: https://docs.jax.dev/en/latest/_autosummary/jax.jit.html
there have been fortran compilers which have done auto parallelization for decades, i think nvidia released a compiler that will take your code and do its best to run it on a gpu
this works best for scientific computing things that run through very big loops where there is very little interaction between iterations
Sure, Tensorflow and Pytorch, here ya go :)
As a user I highly appreciate ./configure for the --help flag, which usually tells me how to build a program with or without particular functionalities which may or may not be applicable to my use-case.
Autoconf can use cache files [1], which can greatly speed up repeated configures. With cache, a test is run at most once.
[1] https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/a...
I get the impression configure not only runs sequentially, but incrementally, where previous results can change the results of tests run later. Were it just sequential, running multiple tests as separate processes would be relatively simple.
Also, you shouldn’t need to run ./configure every time you run make.
No, but if you are doing something like rebuilding a distro's worth of packages from source from scratch, the configure step starts to dominate. I build around 550, and it takes around 6 hours on a single node.
Most checks are common, so what can help is having a shared cache for all configure scripts so if you have 400 packages to rebuild, it doesn't check 400 times if you should use flock or fcntl. This approach is described here: https://jmmv.dev/2022/06/autoconf-caching.html
It doesn't help that autoconf is basically abandonware, with one forlorn maintainer trying to resuscitate it, but creating major regressions with new releases: https://lwn.net/Articles/834682/
Very nice! I always get annoyed when my fancy 16 thread CPU is left barely used as one thread is burning away with the rest sitting and waiting. Bookmarking this for later to play around with whatever projects I use that still use configure.
Also, I was surprised when the animated text at the top of the article wasn't a gif, but actual text. So cool!
CMake also needs this, badly...
Agreed! The CMake Xcode generator is extremely slow because not only is it running the configure tests sequentially, but it generates a new Xcode project for each of them.
Why do we need to even run most of the things in ./configure? Why not just have a file in /etc which is updated when you install various packages which ./configure can read to learn various stats about the environment? Obviously it will still allow setting various things with parameters and create a Makefile, but much faster.
Keep in mind that the build intentionally depends on environment variables, people often install non-packaged dependencies in bad ways, and cross-compiling is a thing, so it's not that simple.
Some relevant discussion/pointers to other notes on this sort of proposal can be found here: https://utcc.utoronto.ca/~cks/space/blog/sysadmin/AutoconfVa...
(The conclusion I distilled out of reading that at the time, I think, was that this is actually sort of happening, but slowly, and autoconf is likely to stick around for a while, if only as a compatibility layer during the transition.)
Not every OS is going to have such a file, and you also don't know if it matches the actual system ./configure runs on.
What I'd like to see is a configure with guarantees that if the configure succeeds, then the build will succeed too.
autotools is a complete disaster. It’s mind boggling to think that everything we build is usually on top of this arcane system
I was really hoping he worked some autoreconf/macro magic to transform existing configure.ac files into a parallelized result.
Nice writeup though.
It is possible in theory to speed up existing configure scripts by switching interpreter from /bin/sh to something that scans file, splits it to independent blocks and runs them in parallel.
Is there any such previous work?
(Luckily?) With c++ your build will nearly always take longer then the configuration step.
rust: "hold my leggings"
Since when? I far more often run into CMake taking ages than Cargo.
>The purpose of a ./configure script is basically to run the compiler a bunch of times and check which runs succeeded.
Wait is this true? (!)
Historically, different Unixes varied a lot more than they do today. Say you want your program to use the C library function foo on platforms where it’s available and the function bar where it isn’t: You can write both versions and choose between them based on a C preprocessor macro, and the program will use the best option available for the platform where it was compiled.
But now the user has to set the preprocessor macro appropriately when he builds your program. Nobody wants to give the user a pop quiz on the intricacies of his C library every time he goes to install new software. So instead the developer writes a shell script that tries to compile a trivial program that uses function foo. If the script succeeds, it defines the preprocessor macro FOO_AVAILABLE, and the program will use foo; if it fails, it doesn’t define that macro, and the program will fall back to bar.
That shell script grew into configure. A configure script for an old and widely ported piece of software can check for a lot of platform features.
I'm not saying we should send everyone a docker container with a full copy of ubuntu, electron and foo.js whether they have foo in their c library or not, but maybe there is a middle ground?
2 replies →
The closer and deeper you look into the C toolchains the more grossed out you’ll be
Hands have to get dirty somewhere. "As deep as The Worker's City lay underground, so high above towered the City of Metropolis."
The choices are:
1. Restrict the freedom of CPU designers to some approximation of the PDP11. No funky DSP chips. No crazy vector processors.
2. Restrict the freedom of OS designers to some approximation of Unix. No bespoke realtime OSes. No research OSes.
3. Insist programmers use a new programming language for these chips and OSes. (This was the case prior to C and Unix.)
4. Insist programmers write in assembly and/or machine code. Perhaps a macro-assembler is acceptable here, but this is inching toward C.
The cost of this flexibility is gross tooling to make it manageable. Can it be done without years and years of accrued M4 and sh? Perhaps, but that's just CMake and CMake is nowhere near as capable as Autotools & friends are when working with legacy platforms.
Yes.
"./configure" has always been the wrong thing for a very long long time. Also slow...
is this really a big deal given you run ./configure once?
it's like systemd trading off non-determinism for boot speed, when it takes 5 minutes to get through the POST
> is this really a big deal given you run ./configure once
I end up running it dozens of times when changing versions, checking out different branches, chasing dependencies.
It’s a big deal.
> it's like systemd trading off non-determinism for boot speed, when it takes 5 minutes to get through the POST
5 minute POST time is a bad analogy. systemd is used in many places, from desktops (that POST quickly) to embedded systems where boot time is critical.
If deterministic boot is important then you would specify it explicitly. Relying on emergent behavior for consistent boot order is bad design.
The number of systems that have 5 minute POST times and need deterministic boot is an edge case of an edge case.
>chasing dependencies.
This aspect of configure, in particular, drives me nuts. Obviously I'd like it to be faster, but it's not the end of the world. I forget what I was trying to build the other week, but I had to make 18 separate runs of configure to find all the things I was missing. When I dug into things it looked like it could probably have done it in 2 runs, each presenting a batch of things that were missing. Instead I got stuck with "configure, install missing package" over and over again.
1 reply →
> to embedded systems where boot time is critical.
if it's critical on an embedded system then you're not running systemd at all
> The number of systems that have 5 minute POST times and need deterministic boot is an edge case of an edge case.
desktop machines are the edge case, there's a LOT more servers running Linux than people using Linux desktops
> Relying on emergent behavior for consistent boot order is bad design.
tell that to the distro authors who 10 years in can't tell the difference between network-online.target, network-pre.target, network.target
2 replies →
> from desktops (that POST quickly)
I take you don't run DDR5?
> I end up running it dozens of times when changing versions, checking out different branches, chasing dependencies.
Yeah... but neither of that is going to change stuff like the size of a data type, the endianness of the architecture you're running on, or the features / build configuration of some library the project depends on.
Parallelization is a bandaid (although a sorely needed!) IMHO, C/C++ libraries desperately need to develop some sort of standard that doesn't require a full gcc build for each tiny test. I'd envision something like nodejs's package.json, just with more specific information about the build details themselves. And for the stuff like datatype sizes, that should be provided by gcc/llvm in a fast-parseable way so that autotools can pick it up.
1 reply →
If you do a lot of bisecting, or bootstrapping, or building compatibility matrices, or really anything that needs you to compile lots of old versions, the repeated ./configure steps really start feeling like a drag.
In a "reasonably well-behaved program", if you have the artifacts from a current configure, like a "config.h" header, they are compatible with older commits, even if configurations changed, as long as the configuration changes were additive: introducing some new test, along with a new symbol in "config.h".
It's possible to skip some of the ./configure steps. Especially for someone who knows the program very well.
1 reply →
For postgresql development, you run configure over and over...
> it's like systemd trading off non-determinism for boot speed, when it takes 5 minutes to get through the POST
That's a bad analogy: if a given deterministic service ordering is needed for a service to correctly start (say because it doesn't start with the systemd unit), it means the non-deterministic systemd service units are not properly encoding the dependencies tree in the Before= and After=
When done properly, both solutions should work the same. However, the solution properly encoding the dependency graph (instead of just projecting it on a 1-dimensional sequence of numbers) will be more flexible: it's the better solution, because it will give you more speed but also more flexibility: you can see the branches any leaf depends on, remove leaves as needed, then cull the useless branches. You could add determinism if you want, but why bother?
It's like using the dependencies of linux packages, and leaving the job of resolving them to package managers (apt, pacman...): you can then remove the useless packages which are no longer required.
Compare that to doing a `make install` of everything to /usr/local in a specific order, as specified by a script: when done properly, both solutions will work, but one solution is clearly better than the other as it encodes more finely the existing dependencies instead of projecting them to a sequence.
You can add determinism if you want to follow a sequence (ex: `apt-get install make` before adding gcc, then add cuda...), or you can use meta package like build-essentials, but being restricted to a sequence gains you nothing.
I don't think it is a bad analogy
given how complicated the boot process is ([1]), and it occurs once a month, I'd rather it was as deterministic as possible
vs. shaving 1% off the boot time
[1]: distros continue to ship subtlety broken unit files, because the model is too complicated
10 replies →
[dead]