Comment by fnordpiglet
20 hours ago
A few years is hardly forever and the state of the world here indicates a lot of low hanging fruit still exists.
An LLM can certainly be cheaper than grep, because it’s an approximation, while a grep is deterministic and must examine every byte in what can be a relatively complex state machine for a regex based grep. There are other scales to consider like the scale of your local hardware vs the highly multitenant and high end hardware of the hyper scale inference providers.
There are already high volume models for coding inference where the reasoning time is crazy low and cheap per token where it can build reasonably simple software so blindingly fast it isn’t implausible the bottleneck is the latency in tools and networks. I find them hard to use at times because I don’t have time to think through the next turn by the time it’s done.
Regardless I wouldn’t be surprised to see a world where tokens are so cheap it’s not worth metering them but charging licensing feels with meter tiers at the far horizons to prevent abuse, charge outliers. Subscription models already set this stage well.
The other side to consider is bountiful capacity will also drive tokens to near zero price. The data center build out is barely underway and as it materializes, as hardware efficiencies improve, as techniques and model science and technology improves, harnesses, methodologies , etc improve, the economics flip from load shedding to trying to keep the data centers utilized. The economics lead to the world where tokens are not a unit of measurement for cost for anyone other than the inference providers to manage their utilization.
> An LLM can certainly be cheaper than grep, because it’s an approximation, while a grep is deterministic and must examine every byte
You mean a grep over terabytes of data vs a LLM with gigabytes of parameters?
If you have so much data, you can use an index to search. It's unlikely that LLMs are going to be cheaper than properly indexed search DBs (which is what we should be comparing them with)
Gigabytes loaded into GPU memory being processed by custom silicon designed for the task that’s exactly the same on every inference? Even now parameters physically baked into the silicon for super low latency? Yes.
The point of a grep is for adhoc analysis of data that isn’t in a database. At best you could use an inverted index, which is extremely expensive to create and maintain, but if you don’t know a priori what you’re going to search for, or even if an inverted index helps with your query, it’s pointless and expensive to index literally everything you could ever use to save some time on a tool call.
The point isn’t to find a way to make searching more efficient. It’s that LLMs are becoming very efficient, and a measure is relative to grep. Making grep and searching more efficient has been a task in CS for decades. LLMs is relatively new beyond a very small crowd. It’s also not a goal to make LLMs better at grep than grep, but as a relative reference point that inference is getting cheap and fast and has opportunity to get faster far beyond grep simply due to unclaimed optimizations more available for the newer technology.