← Back to context

Comment by girvo

7 hours ago

Nah, I’m streaming ngrams off NVMe on my Spark-alike right now. Works surprisingly well (except for when I accidentally bottlenecked it through my NAS)

5 comments

girvo

Reply
jkingsman  6 hours ago

What kind of throughput do you see on what models?

  • girvo  6 hours ago

    GB10 boxes have way more compute than they have memory bandwidth, which nicely fits medium sized MoE models with speculative execution (MTP, DSpark/DFlash, etc)

    Qwen 3.8 Flash Next (what I'm running basically entirely now) sees 30 / 35.0 / 45 tk/s for prose, analysis and code respectively for actual use (not short context benchmarking) with Pi. Thinking blocks are ~35tk/s or so.

    The GB10 having so much compute is great for prefill too, 2000-3000/s for 14k to 64k token prompts (cold cache too) in the quick benchmark I did. 3500tk/s for warm cache which is nice :)

    When I accidentally streamed my ngrams over the 2.5Gb/s network, it cut all the throughput down in half basically. Especially notable for the time-to-first-token, which is what clued me in that I'd messed up somehow!

    For Qwen 3.8 27B, I got it up to a consistent 20tk-25tk/s but 27B thinks so much that it was honestly too painful: Flash Next is as smart, as useful, but much faster for real agentic dev usage IMO

    Laguna S 2.1 saw similar numbers to Flash Next if I remember right, but their latest updates means it doesn't quite fit a GB10 128GB anymore at full context which is a shame.

    Note: these are all NVFP4 quants (usually a dynamic one where some tensor layers are left at full precision though)

    • verdverm  6 hours ago

      I personally stopped caring as much about the tok/s as the agents are largely in the background, and so have also moved preference from MoE to dense

      I want to see about fine-tuning these models a bit on the GB10 to tame that over thinking and some other behaviors (like using tools I don't use)

      qwen 3.8 seems to have been trained with some `rkt` that messes with tool outputs to "save tokens"

verdverm  7 hours ago

interesting, peer comment seems to indicate this is a possibility as well, will have to take a deeper look