Comment by luciana1u
7 hours ago
engineering effort to make something 1000x faster that accounts for 0.1% of total runtime is the most software developer thing imaginable
7 hours ago
engineering effort to make something 1000x faster that accounts for 0.1% of total runtime is the most software developer thing imaginable
"The pursuit of excellence does not need justification."
https://x.com/mitchellh/status/2074225453217505494
I initially had a Rust-based word cloud generator that generates word clouds in high resolution in ~100 milliseconds whereas it would take other generators a couple seconds to do the same thing. Does the world need a super-fast word cloud generator? No. Do I want a super-fast word cloud generator? Yes.
I later found enough optimizations to reduce the generation speed all the way down to ~16ms. Do I need a word cloud generator that fast? No. But if I have a word cloud generator it's going to be as fast as possible dammit.
1000x improvements unlock qualitatively new capabilities, even when only applied to subcomponents. Moreover, the fact that it's only 0.1% of the runtime is usually [0] an artifact of the entire project being treated that way -- why write this the right way when it won't move the needle on the composite project?
[0] Yes, for LLMs we're closer to the bound than 1000x. Even there though, basic pytorch operations are often 2x slower than simple rewrites, better scheduling algorithms have a history of closer to 5x-10x gains, and all of that supposes you don't have further architectural improvements over time. Moreover, who's to say that legitimately fast tokenization doesn't unlock additional capabilities elsewhere which people have ignored because it was never close to viable?
This depends on what your workflow is. There are use cases for tokenization that don't always involve immediately feeding the text into a model.
If you're tokenizing to run a tiny SLM for routing purposes, it can be way more than 0.1%.
This is the "GPU driver optimizations don't matter because PC's sit idle at the desktop most of the time" mindset.
This can make a huge difference for local models on modest hardware
It's pretty funny but then again, why not if it's as trivial to simplify as it appears
1000x faster on 0.1% of runtime = 0.1% saved. Amdahl remains undefeated.
Globally that’s ~50 GWh/yr, or ~5.7 MW continuous:
• 4,700 American homes
• a week of British tea