Comment by nvme0n1p1
1 day ago
Grep (or ripgrep at least) is i/o bottlenecked at this point. It's impossible to process data at faster than i/o speeds, since you have to get the data to the processor somehow. That doesn't change whether that processing is grep on a CPU, or LLM on an ASIC.
ripgrep may be.
I did a toy project once implementing a limited version of grep on an FPGA and was able to get some speedup over GNU grep at the time, though marginal.
In any case, LLMs aren't IO bound :))
I am now imagining a future where developers buy fancy "grep cards" for their machines. I don't hate it.
These are properly called "GPUs", as in "Grep Processing Units"!
It would be cool if they could be daisy-chained, so you can have a hardware implementation of |
13 replies →
I wonder how close you can get with Nvidia's GPUDirect. Hook the fast NVME directly up to the GPU (well, it gets direct DMA to GPU at any rate), then implement parallel grep in CUDA... profit?
I look forward to GPGrepPU -- someone'll find a way to abuse them for scientific workloads or something.
1 reply →
Universal Basic ASICs
1 reply →
At some point Intel was experimenting a Xeon with built-in FPGA, maybe they were just a bit too early to the game :)
1 reply →
I've evaluated some of these and in almost all cases they failed to outperform Intel's ridiculously optimized CPU regex library, hyperscan.
Right, but I read the comment's point as: being i/o bound is the eventual limit of the asymptote and it's the same one grep has.
GPUs and inference ASICS also have large amounts of high bandwidth memory, plus lots of high speed storage cache, and dedicated very high bandwidth scale-out and scale-up networks. Because they are also often bound by I/O bandwidth.
If your problem is grepping crazy amounts of data, the infrastructure for LLMs isn't a bad place to look for an example.
> plus lots of high speed storage cache
I wouldn't be surprised if that's what Apple is focused on for their next generation platforms – I wonder if more layers of caching between their SSDs and unified memory are on the cards.
i/o could still be sped up though? And I dunno, maybe this actually is an argument for how the llm version could end up being faster, because there is lots of investment in crazy fast i/o hardware and protocols to get the data into the chips.