Comment by ycui7

For people with single RTX PRO 6000 96GB or DGX Spark 128GB, vllm-moet is a very good engine, although lesser known. It auto generate a symmetric 2-bit plane for inference and also generate a 4-bit delta cache to recover precision. Support ssd streaming oversized weight. You pick how much VRAM to allocate to each to balance out speed vs precision. 170 tps with ds-v4-flash demonstrated.

It use the stock model, no new models requires.

Worth spend a few hours to try.

The DGX Spark requires a small hack to ignore the difference between sm120 vs sm121, but it does run on sm121.