Comment by zozbot234
4 hours ago
You can definitely offload n-gram embeddings to storage; they're very sparsely used (only a few KB fetched per token) so this is quite effective. Loading to DRAM only becomes necessary if they are a bottleneck to overall performance (which might happen if you're doing very wide batches and everything else uses super fast VRAM/HBM).
I was looking at the qwen-next-flash, and the weights would fill my OEM Spark on their own, before the n-gram. I'm unclear if offloading to disk can work here, is that what you are implying is possible?!
Check out eugr’s TP=1 sparkrun recipe :)
It’s an NVFP4 quant, but it fits, and is surprisingly capable.
do you have a HF link? HF search is not uncovering it for me
(or is it somewhere else)
5 replies →