Comment by sanderjd
1 day ago
Yeah I bumped on that too. If it's possible to make llms cheaper than current grep, then it is also almost certainly possible to make grep cheaper.
1 day ago
Yeah I bumped on that too. If it's possible to make llms cheaper than current grep, then it is also almost certainly possible to make grep cheaper.
You can burn anything* into an ASIC to make it cheaper per-call.
non-backreferencing grep is not very difficult to implement in an ASIC either. But it's probably not worth it because of how relatively rarely you use it and of the data transfer costs.
LLMs are great candidates for ASIC-burning because they're slow compared even to network speeds and run all the time. The issue is that you don't want to burn a specific model or architecture that then becomes obsolete.
So you've got two possible futures, and both guarantee large price drops: (a) LLMs keep getting better and better and better, so ability/$ keeps rising; or (b) LLMs plateau in ability, in which they will start getting ASIC'd.
Grep (or ripgrep at least) is i/o bottlenecked at this point. It's impossible to process data at faster than i/o speeds, since you have to get the data to the processor somehow. That doesn't change whether that processing is grep on a CPU, or LLM on an ASIC.
ripgrep may be.
I did a toy project once implementing a limited version of grep on an FPGA and was able to get some speedup over GNU grep at the time, though marginal.
In any case, LLMs aren't IO bound :))
25 replies →
GPUs and inference ASICS also have large amounts of high bandwidth memory, plus lots of high speed storage cache, and dedicated very high bandwidth scale-out and scale-up networks. Because they are also often bound by I/O bandwidth.
If your problem is grepping crazy amounts of data, the infrastructure for LLMs isn't a bad place to look for an example.
1 reply →
i/o could still be sped up though? And I dunno, maybe this actually is an argument for how the llm version could end up being faster, because there is lots of investment in crazy fast i/o hardware and protocols to get the data into the chips.
It's almost a certainty that LLMs have a "core" that will essentially never become obsolete, possibly even 80% to 90% of their parameters. The rules of English and other languages, core ideas in math and science, all of history, nearly all literature, etc. We don't really understand what's going on inside LLMs enough yet to make good use of this, but one day we will have "core logic" neural networks with stable weights burned into ASIC that are doing the heavy lifting, with more dynamic continually-tuned models manipulating the inputs and outputs into those core models. There are also likely stable expert models on topics that don't change much that we could already do this with.
Inference costs cannot keep falling forever, but they do still have a long way to go.
it doesn't work that way though
> LLMs plateau in ability, in which they will start getting ASIC'd.
They don't need to plateu for that to happen. There are companies already building AI on ASIC, and IIRC they were approach 12 months lead time. A 12 months old frontier model (Sonnet 4.5, GPT-5, Kimi K2) for 1% of the price is still a rather good value proposition.
In order for the ASIC to achieve 1% of the price, there'd have to be 99% overhead in GPU implementations which for some reason you'd have to be able to eliminate in ASICs but not in GPUs. That seems rather implausible.
It will be a better value proposition now than it was 12 months ago. It's likely to be better yet in another 12 months. There may be room for a parallel to Moore's Law here.
2 replies →
To be clear, I agree with the overall premise of the article!
But I would probably take a long horizon bet that the grep implementation on my machine will remain cheaper than an equivalent ai task, even though I think those ai tasks will become far cheaper over time.
I just think the original comment's model of asymptotic approach is probably more likely to be accurate than the model of the line blowing through this grep-like cost level.
This whole story really reminds me of crypto coins. Like.. going from mining one coin, or lets say token, to millions of fractions like 0.00000000001 bitcoin a week.
There is a reason it took 4 and a half billion years for human intelligence to develop.
6 replies →
postgres in asic however...
Do you mean most of uses of grep are backreferencing? In my 30-yr career I did not use regex backrefs once except for learning them.
No, it just gets much more complicated to implement grep both in general, and especially in hardware if you support backreferences, since those make it impossible to compile the regular expression into a state machine.
grep is deterministic. Llm is probabilistic. Llm can be transferred to a tiny quantum cpu or a lower precision float.
When the author wrote Llm can be as cheap as a tool, I read it as not equivalent. They even said the Llm can be embedded into a tool.
Their point was, the higher level use case — like classification — could become as cheap as grep. Which is quite well possible.
Yeah you may be right!
depends on what you are grepping ... greapping a large file might be more expensive one day than generating n-th token with LLM that works fully in hardware
you could make hardware implementation of grep and store the file itself next to it in some ROM but that's not a very useful grep ... while hardware LLM is exactly as useful as software LLM only orders of magnitude faster
Yeah it's a pretty poorly specified problem. It needs to be some kind of "equivalent task", but it's not clear how to define that.