Comment by NitpickLawyer

It is 125B A6B. vLLM is already out with support, ngrams can be offloaded to RAM so you only need ~96GB VRAM for nvfp4 w/ full context.

Likely soon we'll see nvme offloading for ngrams as well. They're just an index, so that should be plenty fast for what it does. LLama.cpp support should come soon as well, and they might do some things with offloading first.