Comment by cyberax
18 hours ago
Buildkit...
It sounds great in theory, but it JustDoesn'tWork(tm).
Its caching is plain broken, and the overhead of transmitting the entire build state to the remote computer every time is just busywork for most cases. I switched to Podman+buildah as a result, because it uses the previous dead simple Docker layered build system.
If you don't believe me, try to make caching work on Github with multi-stage images. Just have a base image and a couple of other images produced from it and try to use the GHA cache to minimize the amount of pulled data.
Never figured out how to get the buildx cache to actually work reliably on ARM OS X. Horrible if you have to build x86 images regularly.
Why would you use the horrible GHA cache and not a much more efficient registry based cache?
Registry cache...
It's yet one more incomprehensible Buildkit decision. The original Docker builder had a very simple cache system: it computed the layer hash and then checked the registry for its presence. Simple content-addressable caching.
Buildkit can NOT do this. Instead, it uses a single image as a dumping ground for the caches. If you have two builders using the same image, they'll step on each other's toes. GHA at least side-steps this.
But I tried the registry cache, and it didn't improve anything. So far, I was not able to get caching to work with multi-stage builds at all. There are open issues for that, dating back to 2020.
How do you use buildah? with dockerfiles?
I find that buildah is sort of unbearably slow when using dockerfiles...
It has a braindead cache checking, I've fixed it locally and I'm cleaning it up for the upstream submission. But otherwise, it's always faster for me than Buildkit.