← Back to context

Comment by stymaar

3 hours ago

n-gram per-layer embeddings[1][2] might be it.

[1] https://sebastianraschka.com/llm-architecture-gallery/per-la...

[2]: See DS 4.1-Flash and Qwen-3.8-Next.

19 comments

stymaar

Reply
verdverm  3 hours ago

this is to offload VRAM to DRAM (for GP comment), and makes no difference for URAM

  • zozbot234  2 hours ago

    You can definitely offload n-gram embeddings to storage; they're very sparsely used (only a few KB fetched per token) so this is quite effective. Loading to DRAM only becomes necessary if they are a bottleneck to overall performance (which might happen if you're doing very wide batches and everything else uses super fast VRAM/HBM).

    • verdverm  2 hours ago

      I was looking at the qwen-next-flash, and the weights would fill my OEM Spark on their own, before the n-gram. I'm unclear if offloading to disk can work here, is that what you are implying is possible?!

      7 replies →

  • girvo  2 hours ago

    Nah, I’m streaming ngrams off NVMe on my Spark-alike right now. Works surprisingly well (except for when I accidentally bottlenecked it through my NAS)

  • petu  2 hours ago

    n-grams can be kept on SSD, no need to hold them in any kind of RAM (at least w/o batching)