I'm not super familiar with Bundler's architecture but I think the most impactful thing would be adopting uv's cache design, which is a big part of what makes uv so fast and should be replicable in other languages and ecosystems.
> Ignoring requires-python upper bounds. When a package says it requires python<4.0, uv ignores the upper bound and only checks the lower. This reduces resolver backtracking dramatically since upper bounds are almost always wrong.
I don't think that ignoring upper bounds has a significant impact on uv's performance. We do this for a totally different reason, which is that it leads to better solves. For example, if you say your project requires Python 3.8 or later, but some dependency said it works for ">=3.8,<4", then suddenly your project isn't installable on Python 4, and you'd be implicitly required to put a "<4" bound on your own project. uv solves for all of your supported Python versions, not a single version, so discounting the upper bounds doesn't actually save us any time in the solve.
> PEP 658 (2022) put package metadata directly in the Simple Repository API, so resolvers could fetch dependency information without downloading wheels at all.
> Fortunately RubyGems.org already provides the same information about gems.
> [...]
> After we unpack the gem, we can discover whether the gem is a native extension or not.
Why not adding the meta information of whether the gem is a native extension or not directly to rubygems.org? You could fully parallelize whole installation trees of dependencies then.
Had the same thought reading this but I suspect what's in the gemspec could accidentally differ from what's in the RubyGems.org metadata, although that should probably not be possible.
From working on RubyGems.org a long time ago I vaguely remember that the metadata extracted from the gemspec is version-specific. So if you add a new native_extension boolean you'd have to artificially reprocess those previously published gemspecs to change the metadata for all past versions.
Being able to mutate metadata for past versions is dangerous enough that I'd be surprised it's allowed or even possible. So that might not even be something Aaron considered here for that reason. That said, it seems reasonable to me to suggest this improvement going forward to make unpacking the gem unnecessary to know whether it'll affect installation order.
I appreciate that Aaron is focusing on the practical algorithm/design improvements that could be made to Bundler, vs. prematurely going all in on "rewrite in Rust".
I manage all my programs via ruby in a manner similar to GoboLinux, e. g. versioned AppDirs. So I don't need uv or anything else for this - not that I have anything against uv, it is just not something I need myself. But I agree with you in that there are multiple objectives or goals to be had here; you mention one, Aaron mentioned speed. I think there are many more (I mentioned documentation before too; people in ruby seem to think documentation is not important, or at the least really many think in that way - look at the documentation of rack, it is virtually not existing. I am not going to sift through low quality ruby code for hours to try to figure out what madness drove this or that author to write some horrible-to-read junk of code. A project without good documentation is worthless. Why do ruby developers not learn this lesson?).
I think all of ruby, including the ecosystem as well as how ruby is governed, needs to be considered. In particular with ruby constantly declining and losing user share. That is not good. That should change.
I'm always surprised to hear this, and I want to be clear that I'm not trying to be dismissive in my comment. However, I've not encountered issues while juggling dozens of Ruby projects since around 2011, despite seeing many people's complaints over the years. Ten years ago I was using rvm, and I saw people sharing their issues with it, and listing reasons why rbenv and chruby are better. So I tried those, and my resulting workflow felt basically the same once I got used to the differences.
At this point I've used rbenv, rvm, asdf, mise, and one other whose name isn't coming to mind. Not to mention docker containers, with or without any of those tools.
I don't mean to project any particular complaint onto you, and I'm curious what part of it is infuriating? Each of the version managers I've used has functioned as advertised, and I'm able to get back to work pretty smoothly.
what exactly is your issue? I've been using rvm for a decade(?) without any major pain. Cross-language tools such as mise or asdf also seem to work ok.
I can relate to the "I wish we didn't need a second tool", but it doesn't seem like much of a mess.
Hmmm. Aaron is cool, but also works at Shopify. Neither DHH nor Aaron mentioned anyone at the gem.coop project. I can't help but have mixed feelings here.
I think the underlying issue is much bigger than merely contained on speed. It also has to do with control - both for developers as well as users.
This is also why I think the label "prematurely" is weird, because it attempts to singularize everything down to the speed label. I think the issue is much wider than what is faster. In fact, speed doesn't interest me that much; whether it takes 0.35 seconds or 0.45 seconds or even 1.5 seconds to install a gem, is just so irrelevant to me. But, for instance, high quality documentation - now that is an area where ruby failed in epic ways. Not all projects but many. And then ruby wonders why it is dying? And nobody wants to focus on these issues really. So the issue should really be viewed much larger in scope than "merely" "speed is the issue". I mean ... matz focused on speed in the last 10 years. Ok. Ruby declined. Well, perhaps it is time to focus on other things more - not that people mind a faster ruby, but if a language is dying, then you need to think really hard, come up with a plan, multiple ideas, and push these hard. Rather than, say, purge away old developers and claim nothing happened here and all is well ...
I don't know a lot about Ruby, but I'd wager what its missing is a hero app or framework. Ruby on Rails got folks interested for awhile, but I guess other frameworks won out. What left does it have? What domains does it excel at?
Python has ML. JS has web. C/C++ has performance. Rust is stealing a slice of that thanks to safety.
That probably covers like 99% of things, at least from my world view. There are arguably other better languages but it doesn't much matter if the community all flocks to the well established ones.
The biggest thing that gems could do to make rubygems faster is to have a registry/database of files for each gem, so that rubygems didn't have to scan the filesystem on every `require` looking for which gem had which file in it.
That would mean that if you edited your gems directly, things would break. Add a file, and it wouldn't get found until the metadata got rehashed. The gem install, uninstall, etc commands would need to be modified to maintain that metadata. But really, you shouldn't be hacking up your gem library like that ith shellcommands anyway (and if you are doing manual surgery, having to regen the metadata isn't really that burdensome).
At runtime rather than install time yes. I did some prototyping on this back in the day, one of the issues is the language lacks an efficient data structure to store that information and you can’t (easily) build one efficiently because instances are too heavy.
I wrote some code to do almost this many years ago (if I recall correctly, it doesn’t cache anything to disk, but builds the hash fresh each time, which can still result in massive speed up).
Probably obsolete and broken by now, but one of my favorite mini projects.
(And I just realized the graph is all but impossible to read in dark mode)
100%. Optimising "bundle install" etc. is optimizing the wrong thing. You don't even need this to work from gems in general. It'd have solved a lot of problems just to have it work for "bundle install" in standalone mode, where all the files are installed to a directory anyway.
But in general, one of the biggest problems with Ruby for me is how $LOAD_PATH causes combinatoric explosion when you add gems because every gem is added, due to the lack of any scoping of require's to packages.
The existence of multiple projects to cache this is an illustration that this is a real issue. I've had project in the past where starting the app took minutes purely due to require's, and where we shaved minutes off by crude manipulation of the load path, as most of that time was pointless stat calls.
Really interesting post, but this part from the beginning stuck out to me:
Ruby Gems are tar files, and one of the files in the tar file is a YAML representation of the GemSpec. This YAML file declares all dependencies for the Gem, so RubyGems can know, without evaling anything, what dependencies it needs to install before it can install any particular Gem. Additionally, RubyGems.org provides an API for asking about dependency information, which is actually the normal way of getting dependency info (again, no eval required).
It would be interesting to compare and contrast the parsing speed for a large representative set of Python dependencies compared to a large representative set of Ruby dependencies. YAML is famously not the most efficient format to parse. We might have been better than `pip`, but I would be surprised if there isn't any room left on the table to parse dependency information in a more efficient format (JSON, protobufs, whatever).
That said, the points at the end about not needing to parse gemspecs to install "most" dependencies would make this pretty moot (if the information is already returned from the gemserver)
Although Yaml is a dreadful thing, given the context and the size of a normal gemspec I would be very surprised if it showed up in any significant capacity when psych should be in the low single digit MB/s throughput.
For a "YAML" lockfile, you could probably write a much simpler and much more performant parser that throws out much of what makes YAML complicated, in particular, anchors, data type tags, all the ways of doing multi-line strings, all the weird unexpected type conversions (like yes/no converting to a boolean)... If the lockfile is never meant to be edited by human hands, only reviewed by human eyes, you can build a much simpler parser for something like:
It mostly doesn't matter, because these metadata files are pulled into their respective package managers. When you publish to RubyGems the file is read into their database and made available to their API, just like when you publish a Python file the pyproject.toml is parse into the PyPI database and made available.
This is a major reason why UV is faster than older python package managers, as they were able to take advantage of the change in the PyPI registry that enabled this. Now these package managers can run their dependency calculations without needing to download the entire package, decompress the package files, and then parse them.
I think fibers (or, rather, the Async library) in Ruby tends to be fetishized by junior Rails engineers who don't realize higher level thread coordination issues (connection pools, etc) equally apply to fibers. That said this could be a pretty good use case for fibers -- the code base I use every day has ~230 gems and if you can peel off the actual IO bound installation of all those into non-blocking calls, you would see a meaningful performance difference vs spinning up threads and context switching between them.
What I would do to really squeeze the rest out in pure ruby (bear in mind I’ve been away about a decade so there _might be_ new bits but nothing meaningful as far as I know):
Use a cheaper to parse index format (the gists I wrote years ago cover this: https://gist.github.com/raggi/4957402)
Use threads for the initial archive downloads (this is just io, and you want to reuse some caches like the index)
Use a few forks for the unpacking and post install steps (because these have unpredictable concurrency behaviors)
> there _might be_ new bits but nothing meaningful as far as I know
If you didn't need backwards compatibility with older rubies you could use Ractors in lieu of forks and not have to IPC between the two and have cleaner communication channels. I can peg all the cores on my machine with a simple Ractor pool doing simple computation, which feels like a miracle as a Ruby old head. Bundler could get away with creating their own Ractor safe installer pool which would be cool as it'd be the first large scale use of Ractors that I know of.
I’ve been squinting at the “global cache for all bundler instances” issue[1] and I’m trying to figure out if it’s a minefield of hidden complication or if it’s actually relatively straight forward.
It’s interesting as a target because it pays off more the longer it has been implemented as it only would be shared from versions going forward.
It's definitely not super straightforward, but there's plenty of recent prior art to steal from. Ruby was probably not the best place to solve this for the first time given the constraints (similar to pip), but there's no reason the Ruby ecosystem shouldn't now benefit from the work other ecosystems have done to solve it.
Rails 8.1 and ruby 3 are also very surprisingly fast, and coming back to an “omakase” framework is honestly a breath of fresh air especially now that with AI tools you can implement a lot of stuff from scratch instead of using deps.
Well, now my opinion of uv has been damaged. It...
> Ignoring requires-python upper bounds. When a package says it requires python<4.0, uv ignores the upper bound and only checks the lower. This reduces resolver backtracking dramatically since upper bounds are almost always wrong. Packages declare python<4.0 because they haven’t tested on Python 4, not because they’ll actually break. The constraint is defensive, not predictive
Man, it's easy to be fast when you're wrong. But of course it is fast because Rust not because it just skips the hard parts of dependency constraint solving and hopes people don't notice.
> When multiple package indexes are configured, pip checks all of them. uv picks from the first index that has the package, stopping there. This prevents dependency confusion attacks and avoids extra network requests.
Ambiguity detection is important.
> uv ignores pip’s configuration files entirely. No parsing, no environment variable lookups, no inheritance from system-wide and per-user locations.
Stuff like this sense unlikely to contribute to overall runtime, but it does decrease flexibility.
> No bytecode compilation by default. pip compiles .py files to .pyc during installation. uv skips this step, shaving time off every install.
... thus shifting the bytecode compilation burden to first startup after install. You're still paying for the bytecode compilation (and it's serialized, so you're actually spending more time), but you don't associate the time with your package manager.
I mean, sure, avoiding tons of Python subprocesses helps, but in our bold new free threaded world, we don't have to spawn so many subprocesses.
>> Ignoring requires-python upper bounds. When a package says it requires python<4.0, uv ignores the upper bound and only checks the lower. This reduces resolver backtracking dramatically since upper bounds are almost always wrong. Packages declare python<4.0 because they haven’t tested on Python 4, not because they’ll actually break. The constraint is defensive, not predictive
>
> Man, it's easy to be fast when you're wrong. But of course it is fast because Rust not because it just skips the hard parts of dependency constraint solving and hopes people don't notice.
Version bound checking is NP complete but becomes tractable by dropping the upper bound constraint. Russ Cox researched version selection in 2016 and described the problem in his "Version SAT" blog post (https://research.swtch.com/version-sat). This research is what informed Go's Minimal Version Selection (https://research.swtch.com/vgo-mvs) for modules.
It appears to me that uv is walking the same path. If most developers don't care about upper bounds and we can avoid expensive algorithms that may never converge, then dropping upper bound support is reasonable. And if uv becomes popular, then it'll be a sign that perhaps Python's ecosystem as a whole will drop package version upper bounds.
Perhaps so, although I'm more algorithmically optimistic. If ignoring upper bounds makes the problem more tractable, you can
1. solve dependency constraints as if upper bounds were absent,
2. check that your solution actually satisfies constraints (O(N), quick and passes almost all the time), and then
3. only if the upper bound constraint check fails, fall back to the slower and reliable parser.
This approach would be clever, efficient, and correct. What you don't get to do is just ignore the fucking rules to which another system studiously adheres then claim you're faster than that system.
> Man, it's easy to be fast when you're wrong. But of course it is fast because Rust not because it just skips the hard parts of dependency constraint solving and hopes people don't notice.
What's underhanded about this? What are the observable effects of this choice that make it wrong? They reformulated the a problem into a different problem that they could solve faster, and then solved that, and got away with it. Sounds like creative problem solving to me.
> Man, it's easy to be fast when you're wrong. But of course it is fast because Rust not because it just skips the hard parts of dependency constraint solving and hopes people don't notice.
I think you're misunderstanding why we do this: it's a security feature. pip's design is inherently vulnerable to dependency confusion attacks, since packages of the same name across indexes are considered equally trusted by pip. You can look up the torchtriton attack to learn more.
> Stuff like this sense unlikely to contribute to overall runtime, but it does decrease flexibility.
I think you're misinformed. We support all of these features: system- and per-user configuration files, environment variables, etc. We just don't read _pip's_ configuration file, which is intended for pip, not uv.
> uv ignores pip’s configuration files entirely. No parsing, no environment variable lookups, no inheritance from system-wide and per-user locations.
Stuff like this sense unlikely to contribute to overall runtime, but it does decrease flexibility.
Astral have been very clear that they have no intention of replicating all of pip. uv pip install was a way to smooth the transition from using pip to using uv. The point of uv wasn't to rewrite pip in rust - and thankfully so. For all of the good that pip did it has shortcomings which only a new package manager turned out capable of solving.
> No bytecode compilation by default. pip compiles .py files to .pyc during installation. uv skips this step, shaving time off every install.
... thus shifting the bytecode compilation burden to first startup after install. You're still paying for the bytecode compilation (and it's serialized, so you're actually spending more time), but you don't associate the time with your package manager.
In most cases this will have no noticeable impact (so a sane default) - but when it does count you simply turn on --compile-bytecode.
I agree that bytecode compilation (and caching to pyc files) seldom has a meaningful impact, but it's nevertheless unfair to tout it as an advantage of uv over pip, because by doing so you've constructed an apples to oranges comparison.
You could argue that uv has a better default behavior than pip, but that's not an engineering advantage: it's just a different choice of default setting. If you turned off eager bytecode compilation in pip you'd get the same result.
There's never going to be a Python 4 so I don't think they are wrong. Even if lighting strikes thrice there's no way they could migrate people to Python 4 before uv could be updated to "fix" that.
> Ambiguity detection is important.
I'm not sure what you mean here. Pip doesn't detect any ambiguities. In fact Pip's behaviour is a gaping security hole that they've refused to fix, and as far as I know the only way to avoid it is to use `uv` (or register all of your internal company package names on PyPI which nobody wants to do).
> thus shifting the bytecode compilation burden to first startup after install
> Pip doesn't detect any ambiguities. In fact Pip's behaviour is a gaping security hole that they've refused to fix, and as far as I know the only way to avoid it is to use `uv`
Agreed the current behavior is stupid, FWIW. I hope PEPs 708 and 752 get implemented soon. I'm just pointing out that there's an important qualitative difference between
1. we do the same job, but much faster; and
2. we decided your job is stupid and so don't do it, realizing speedups.
uv presents itself as #1 but is actually #2, and that's a shame.
I've been doing software of all kinds for a long long time.
I've never, ever, been in a position where I was concerned about the speed of my package manager.
When you're in a big company, the problem really starts showing. A service can have 100+ dependencies, many in private repos, and once you start adding and modifying dependencies, it has to figure out how to version it to create the lock file across all of these and it can be really slow.
Cloud dev environments can also take several minutes to set up.
Many of these package managers get invoked countless times per day (e.g., in CI to prepare an environment and run tests, while spinning up new dev/AI agent environments, etc).
Is the package manager a significant amount of time compared to setting up containers, running tests etc? (Genuine question, I’m on holiday and can’t look up real stats for myself right now)
Ye,s but if your CI isn't terrible, you have the dependencies cached, so that subsequent runs are almost instant, and more importantly, you don't have a hard dependency on a third party service.
The reason for speeding up bundler isn't CI, it's newcomer experience. `bundle install` is the overwhelming majority of the duration of `rails new`.
I'm not super familiar with Bundler's architecture but I think the most impactful thing would be adopting uv's cache design, which is a big part of what makes uv so fast and should be replicable in other languages and ecosystems.
> Ignoring requires-python upper bounds. When a package says it requires python<4.0, uv ignores the upper bound and only checks the lower. This reduces resolver backtracking dramatically since upper bounds are almost always wrong.
I don't think that ignoring upper bounds has a significant impact on uv's performance. We do this for a totally different reason, which is that it leads to better solves. For example, if you say your project requires Python 3.8 or later, but some dependency said it works for ">=3.8,<4", then suddenly your project isn't installable on Python 4, and you'd be implicitly required to put a "<4" bound on your own project. uv solves for all of your supported Python versions, not a single version, so discounting the upper bounds doesn't actually save us any time in the solve.
(See, e.g. https://discuss.python.org/t/requires-python-upper-limits/12....)
> PEP 658 (2022) put package metadata directly in the Simple Repository API, so resolvers could fetch dependency information without downloading wheels at all.
> Fortunately RubyGems.org already provides the same information about gems.
> [...]
> After we unpack the gem, we can discover whether the gem is a native extension or not.
Why not adding the meta information of whether the gem is a native extension or not directly to rubygems.org? You could fully parallelize whole installation trees of dependencies then.
Had the same thought reading this but I suspect what's in the gemspec could accidentally differ from what's in the RubyGems.org metadata, although that should probably not be possible.
From working on RubyGems.org a long time ago I vaguely remember that the metadata extracted from the gemspec is version-specific. So if you add a new native_extension boolean you'd have to artificially reprocess those previously published gemspecs to change the metadata for all past versions.
Being able to mutate metadata for past versions is dangerous enough that I'd be surprised it's allowed or even possible. So that might not even be something Aaron considered here for that reason. That said, it seems reasonable to me to suggest this improvement going forward to make unpacking the gem unnecessary to know whether it'll affect installation order.
I appreciate that Aaron is focusing on the practical algorithm/design improvements that could be made to Bundler, vs. prematurely going all in on "rewrite in Rust".
Speed would be nice, but more than that I want it to also manage Ruby installs. I’m infuriated at the mess of Rubys and version managers.
Mise is the answer to this. I no longer use chruby/rbenv/rvm. And it manages multiple languages, project-local environment, etc.
10 replies →
I manage all my programs via ruby in a manner similar to GoboLinux, e. g. versioned AppDirs. So I don't need uv or anything else for this - not that I have anything against uv, it is just not something I need myself. But I agree with you in that there are multiple objectives or goals to be had here; you mention one, Aaron mentioned speed. I think there are many more (I mentioned documentation before too; people in ruby seem to think documentation is not important, or at the least really many think in that way - look at the documentation of rack, it is virtually not existing. I am not going to sift through low quality ruby code for hours to try to figure out what madness drove this or that author to write some horrible-to-read junk of code. A project without good documentation is worthless. Why do ruby developers not learn this lesson?).
I think all of ruby, including the ecosystem as well as how ruby is governed, needs to be considered. In particular with ruby constantly declining and losing user share. That is not good. That should change.
I'm always surprised to hear this, and I want to be clear that I'm not trying to be dismissive in my comment. However, I've not encountered issues while juggling dozens of Ruby projects since around 2011, despite seeing many people's complaints over the years. Ten years ago I was using rvm, and I saw people sharing their issues with it, and listing reasons why rbenv and chruby are better. So I tried those, and my resulting workflow felt basically the same once I got used to the differences.
At this point I've used rbenv, rvm, asdf, mise, and one other whose name isn't coming to mind. Not to mention docker containers, with or without any of those tools.
I don't mean to project any particular complaint onto you, and I'm curious what part of it is infuriating? Each of the version managers I've used has functioned as advertised, and I'm able to get back to work pretty smoothly.
2 replies →
Maybe https://github.com/spinel-coop/rv will be the answer when it's ready
what exactly is your issue? I've been using rvm for a decade(?) without any major pain. Cross-language tools such as mise or asdf also seem to work ok.
I can relate to the "I wish we didn't need a second tool", but it doesn't seem like much of a mess.
2 replies →
What do you use to manage other languages? Asking because asdf is basically a multi language version of a ruby version manager rbenv
Hmmm. Aaron is cool, but also works at Shopify. Neither DHH nor Aaron mentioned anyone at the gem.coop project. I can't help but have mixed feelings here.
I think the underlying issue is much bigger than merely contained on speed. It also has to do with control - both for developers as well as users.
This is also why I think the label "prematurely" is weird, because it attempts to singularize everything down to the speed label. I think the issue is much wider than what is faster. In fact, speed doesn't interest me that much; whether it takes 0.35 seconds or 0.45 seconds or even 1.5 seconds to install a gem, is just so irrelevant to me. But, for instance, high quality documentation - now that is an area where ruby failed in epic ways. Not all projects but many. And then ruby wonders why it is dying? And nobody wants to focus on these issues really. So the issue should really be viewed much larger in scope than "merely" "speed is the issue". I mean ... matz focused on speed in the last 10 years. Ok. Ruby declined. Well, perhaps it is time to focus on other things more - not that people mind a faster ruby, but if a language is dying, then you need to think really hard, come up with a plan, multiple ideas, and push these hard. Rather than, say, purge away old developers and claim nothing happened here and all is well ...
I don't know a lot about Ruby, but I'd wager what its missing is a hero app or framework. Ruby on Rails got folks interested for awhile, but I guess other frameworks won out. What left does it have? What domains does it excel at?
Python has ML. JS has web. C/C++ has performance. Rust is stealing a slice of that thanks to safety.
That probably covers like 99% of things, at least from my world view. There are arguably other better languages but it doesn't much matter if the community all flocks to the well established ones.
2 replies →
The biggest thing that gems could do to make rubygems faster is to have a registry/database of files for each gem, so that rubygems didn't have to scan the filesystem on every `require` looking for which gem had which file in it.
That would mean that if you edited your gems directly, things would break. Add a file, and it wouldn't get found until the metadata got rehashed. The gem install, uninstall, etc commands would need to be modified to maintain that metadata. But really, you shouldn't be hacking up your gem library like that ith shellcommands anyway (and if you are doing manual surgery, having to regen the metadata isn't really that burdensome).
At runtime rather than install time yes. I did some prototyping on this back in the day, one of the issues is the language lacks an efficient data structure to store that information and you can’t (easily) build one efficiently because instances are too heavy.
I wrote some code to do almost this many years ago (if I recall correctly, it doesn’t cache anything to disk, but builds the hash fresh each time, which can still result in massive speed up).
Probably obsolete and broken by now, but one of my favorite mini projects.
(And I just realized the graph is all but impossible to read in dark mode)
https://github.com/pmahoney/fastup
100%. Optimising "bundle install" etc. is optimizing the wrong thing. You don't even need this to work from gems in general. It'd have solved a lot of problems just to have it work for "bundle install" in standalone mode, where all the files are installed to a directory anyway.
But in general, one of the biggest problems with Ruby for me is how $LOAD_PATH causes combinatoric explosion when you add gems because every gem is added, due to the lack of any scoping of require's to packages.
The existence of multiple projects to cache this is an illustration that this is a real issue. I've had project in the past where starting the app took minutes purely due to require's, and where we shaved minutes off by crude manipulation of the load path, as most of that time was pointless stat calls.
You are describing bootsnap.
And yes I proposed to integrate bootsnap into bundler ages ago, but got told to go away.
Recent and related:
How uv got so fast - https://news.ycombinator.com/item?id=46393992 - Dec 2025 (457 comments)
Really interesting post, but this part from the beginning stuck out to me:
It would be interesting to compare and contrast the parsing speed for a large representative set of Python dependencies compared to a large representative set of Ruby dependencies. YAML is famously not the most efficient format to parse. We might have been better than `pip`, but I would be surprised if there isn't any room left on the table to parse dependency information in a more efficient format (JSON, protobufs, whatever).
That said, the points at the end about not needing to parse gemspecs to install "most" dependencies would make this pretty moot (if the information is already returned from the gemserver)
Although Yaml is a dreadful thing, given the context and the size of a normal gemspec I would be very surprised if it showed up in any significant capacity when psych should be in the low single digit MB/s throughput.
For a "YAML" lockfile, you could probably write a much simpler and much more performant parser that throws out much of what makes YAML complicated, in particular, anchors, data type tags, all the ways of doing multi-line strings, all the weird unexpected type conversions (like yes/no converting to a boolean)... If the lockfile is never meant to be edited by human hands, only reviewed by human eyes, you can build a much simpler parser for something like:
It mostly doesn't matter, because these metadata files are pulled into their respective package managers. When you publish to RubyGems the file is read into their database and made available to their API, just like when you publish a Python file the pyproject.toml is parse into the PyPI database and made available.
This is a major reason why UV is faster than older python package managers, as they were able to take advantage of the change in the PyPI registry that enabled this. Now these package managers can run their dependency calculations without needing to download the entire package, decompress the package files, and then parse them.
It’s definitely possible, I wrote a prototype many many years ago https://ra66i.org/tmp/how_gems_should_be.mov
I think fibers (or, rather, the Async library) in Ruby tends to be fetishized by junior Rails engineers who don't realize higher level thread coordination issues (connection pools, etc) equally apply to fibers. That said this could be a pretty good use case for fibers -- the code base I use every day has ~230 gems and if you can peel off the actual IO bound installation of all those into non-blocking calls, you would see a meaningful performance difference vs spinning up threads and context switching between them.
What I would do to really squeeze the rest out in pure ruby (bear in mind I’ve been away about a decade so there _might be_ new bits but nothing meaningful as far as I know): Use a cheaper to parse index format (the gists I wrote years ago cover this: https://gist.github.com/raggi/4957402) Use threads for the initial archive downloads (this is just io, and you want to reuse some caches like the index) Use a few forks for the unpacking and post install steps (because these have unpredictable concurrency behaviors)
> there _might be_ new bits but nothing meaningful as far as I know
If you didn't need backwards compatibility with older rubies you could use Ractors in lieu of forks and not have to IPC between the two and have cleaner communication channels. I can peg all the cores on my machine with a simple Ractor pool doing simple computation, which feels like a miracle as a Ruby old head. Bundler could get away with creating their own Ractor safe installer pool which would be cool as it'd be the first large scale use of Ractors that I know of.
Optimization: Doing nothing is faster than doing something.
Do smart thing make code go brr? No. Stop do dumb thing make code go brr.
I’ve been squinting at the “global cache for all bundler instances” issue[1] and I’m trying to figure out if it’s a minefield of hidden complication or if it’s actually relatively straight forward.
It’s interesting as a target because it pays off more the longer it has been implemented as it only would be shared from versions going forward.
[1] https://github.com/ruby/rubygems/issues/7249
It's definitely not super straightforward, but there's plenty of recent prior art to steal from. Ruby was probably not the best place to solve this for the first time given the constraints (similar to pip), but there's no reason the Ruby ecosystem shouldn't now benefit from the work other ecosystems have done to solve it.
I never found Bundler to be all that slow compared to other package managers.
Rails 8.1 and ruby 3 are also very surprisingly fast, and coming back to an “omakase” framework is honestly a breath of fresh air especially now that with AI tools you can implement a lot of stuff from scratch instead of using deps.
I wouldn't say it stands out as being unusually slow, but it certainly isn't snappy.
tenderlove never misses
Well, now my opinion of uv has been damaged. It...
> Ignoring requires-python upper bounds. When a package says it requires python<4.0, uv ignores the upper bound and only checks the lower. This reduces resolver backtracking dramatically since upper bounds are almost always wrong. Packages declare python<4.0 because they haven’t tested on Python 4, not because they’ll actually break. The constraint is defensive, not predictive
Man, it's easy to be fast when you're wrong. But of course it is fast because Rust not because it just skips the hard parts of dependency constraint solving and hopes people don't notice.
> When multiple package indexes are configured, pip checks all of them. uv picks from the first index that has the package, stopping there. This prevents dependency confusion attacks and avoids extra network requests.
Ambiguity detection is important.
> uv ignores pip’s configuration files entirely. No parsing, no environment variable lookups, no inheritance from system-wide and per-user locations.
Stuff like this sense unlikely to contribute to overall runtime, but it does decrease flexibility.
> No bytecode compilation by default. pip compiles .py files to .pyc during installation. uv skips this step, shaving time off every install.
... thus shifting the bytecode compilation burden to first startup after install. You're still paying for the bytecode compilation (and it's serialized, so you're actually spending more time), but you don't associate the time with your package manager.
I mean, sure, avoiding tons of Python subprocesses helps, but in our bold new free threaded world, we don't have to spawn so many subprocesses.
>> Ignoring requires-python upper bounds. When a package says it requires python<4.0, uv ignores the upper bound and only checks the lower. This reduces resolver backtracking dramatically since upper bounds are almost always wrong. Packages declare python<4.0 because they haven’t tested on Python 4, not because they’ll actually break. The constraint is defensive, not predictive > > Man, it's easy to be fast when you're wrong. But of course it is fast because Rust not because it just skips the hard parts of dependency constraint solving and hopes people don't notice.
Version bound checking is NP complete but becomes tractable by dropping the upper bound constraint. Russ Cox researched version selection in 2016 and described the problem in his "Version SAT" blog post (https://research.swtch.com/version-sat). This research is what informed Go's Minimal Version Selection (https://research.swtch.com/vgo-mvs) for modules.
It appears to me that uv is walking the same path. If most developers don't care about upper bounds and we can avoid expensive algorithms that may never converge, then dropping upper bound support is reasonable. And if uv becomes popular, then it'll be a sign that perhaps Python's ecosystem as a whole will drop package version upper bounds.
Perhaps so, although I'm more algorithmically optimistic. If ignoring upper bounds makes the problem more tractable, you can
1. solve dependency constraints as if upper bounds were absent,
2. check that your solution actually satisfies constraints (O(N), quick and passes almost all the time), and then
3. only if the upper bound constraint check fails, fall back to the slower and reliable parser.
This approach would be clever, efficient, and correct. What you don't get to do is just ignore the fucking rules to which another system studiously adheres then claim you're faster than that system.
That's called cheating.
5 replies →
> Man, it's easy to be fast when you're wrong. But of course it is fast because Rust not because it just skips the hard parts of dependency constraint solving and hopes people don't notice.
What's underhanded about this? What are the observable effects of this choice that make it wrong? They reformulated the a problem into a different problem that they could solve faster, and then solved that, and got away with it. Sounds like creative problem solving to me.
> Man, it's easy to be fast when you're wrong. But of course it is fast because Rust not because it just skips the hard parts of dependency constraint solving and hopes people don't notice.
We ignore upper bounds because it leads to a better solve. You can read my comment here: https://discuss.python.org/t/requires-python-upper-limits/12....
> Ambiguity detection is important.
I think you're misunderstanding why we do this: it's a security feature. pip's design is inherently vulnerable to dependency confusion attacks, since packages of the same name across indexes are considered equally trusted by pip. You can look up the torchtriton attack to learn more.
> Stuff like this sense unlikely to contribute to overall runtime, but it does decrease flexibility.
I think you're misinformed. We support all of these features: system- and per-user configuration files, environment variables, etc. We just don't read _pip's_ configuration file, which is intended for pip, not uv.
> uv ignores pip’s configuration files entirely. No parsing, no environment variable lookups, no inheritance from system-wide and per-user locations.
Stuff like this sense unlikely to contribute to overall runtime, but it does decrease flexibility.
Astral have been very clear that they have no intention of replicating all of pip. uv pip install was a way to smooth the transition from using pip to using uv. The point of uv wasn't to rewrite pip in rust - and thankfully so. For all of the good that pip did it has shortcomings which only a new package manager turned out capable of solving.
> No bytecode compilation by default. pip compiles .py files to .pyc during installation. uv skips this step, shaving time off every install.
... thus shifting the bytecode compilation burden to first startup after install. You're still paying for the bytecode compilation (and it's serialized, so you're actually spending more time), but you don't associate the time with your package manager.
In most cases this will have no noticeable impact (so a sane default) - but when it does count you simply turn on --compile-bytecode.
I agree that bytecode compilation (and caching to pyc files) seldom has a meaningful impact, but it's nevertheless unfair to tout it as an advantage of uv over pip, because by doing so you've constructed an apples to oranges comparison.
You could argue that uv has a better default behavior than pip, but that's not an engineering advantage: it's just a different choice of default setting. If you turned off eager bytecode compilation in pip you'd get the same result.
1 reply →
> Man, it's easy to be fast when you're wrong.
There's never going to be a Python 4 so I don't think they are wrong. Even if lighting strikes thrice there's no way they could migrate people to Python 4 before uv could be updated to "fix" that.
> Ambiguity detection is important.
I'm not sure what you mean here. Pip doesn't detect any ambiguities. In fact Pip's behaviour is a gaping security hole that they've refused to fix, and as far as I know the only way to avoid it is to use `uv` (or register all of your internal company package names on PyPI which nobody wants to do).
> thus shifting the bytecode compilation burden to first startup after install
Which is a much better option.
There's never going to be a Python 4
There will, but called Pyku ...
1 reply →
> Pip doesn't detect any ambiguities. In fact Pip's behaviour is a gaping security hole that they've refused to fix, and as far as I know the only way to avoid it is to use `uv`
Agreed the current behavior is stupid, FWIW. I hope PEPs 708 and 752 get implemented soon. I'm just pointing out that there's an important qualitative difference between
1. we do the same job, but much faster; and
2. we decided your job is stupid and so don't do it, realizing speedups.
uv presents itself as #1 but is actually #2, and that's a shame.
1 reply →
I've been doing software of all kinds for a long long time. I've never, ever, been in a position where I was concerned about the speed of my package manager.
Compiler, yes. Linker, sure. Package downloader. No.
When you're in a big company, the problem really starts showing. A service can have 100+ dependencies, many in private repos, and once you start adding and modifying dependencies, it has to figure out how to version it to create the lock file across all of these and it can be really slow.
Cloud dev environments can also take several minutes to set up.
Many of these package managers get invoked countless times per day (e.g., in CI to prepare an environment and run tests, while spinning up new dev/AI agent environments, etc).
Is the package manager a significant amount of time compared to setting up containers, running tests etc? (Genuine question, I’m on holiday and can’t look up real stats for myself right now)
1 reply →
Ye,s but if your CI isn't terrible, you have the dependencies cached, so that subsequent runs are almost instant, and more importantly, you don't have a hard dependency on a third party service.
The reason for speeding up bundler isn't CI, it's newcomer experience. `bundle install` is the overwhelming majority of the duration of `rails new`.
8 replies →
There is no situation where toolchain improvements or workflow improvements should be snuffed at.
It can be harder to justify in private tooling where you might only have a few dozen or hundred devs saving those seconds per each invocation.
But in public tooling, where the benefit is across tens of thousands or more? It's basically always worth it.
2 replies →
Must be nice not to use Python!
I agree for my own projects where the code and libraries are not enormous. The speed and size gains aren’t going to be something that matters enough.
try conda
took an hour to install 30 dependencies
If any Bundler replacement isn't using Minimal Version Selection, I'm not interested.