Comment by nextaccountic
15 hours ago
> An LLM can certainly be cheaper than grep, because it’s an approximation, while a grep is deterministic and must examine every byte
You mean a grep over terabytes of data vs a LLM with gigabytes of parameters?
If you have so much data, you can use an index to search. It's unlikely that LLMs are going to be cheaper than properly indexed search DBs (which is what we should be comparing them with)
Gigabytes loaded into GPU memory being processed by custom silicon designed for the task that’s exactly the same on every inference? Even now parameters physically baked into the silicon for super low latency? Yes.
The point of a grep is for adhoc analysis of data that isn’t in a database. At best you could use an inverted index, which is extremely expensive to create and maintain, but if you don’t know a priori what you’re going to search for, or even if an inverted index helps with your query, it’s pointless and expensive to index literally everything you could ever use to save some time on a tool call.
The point isn’t to find a way to make searching more efficient. It’s that LLMs are becoming very efficient, and a measure is relative to grep. Making grep and searching more efficient has been a task in CS for decades. LLMs is relatively new beyond a very small crowd. It’s also not a goal to make LLMs better at grep than grep, but as a relative reference point that inference is getting cheap and fast and has opportunity to get faster far beyond grep simply due to unclaimed optimizations more available for the newer technology.