Comment by cmrdporcupine

fwiw, w/ gemma4 -- non-diffusion -- I get about 170ms for a single question -> answer and then an additional ~33ms on adding more. While I see people reporting 300ms for this vLLM PR on same hardware (Spark.)

So I don't see the advantage to their approach until you're up beyond 6 or 7 questions?

Latest commits added gemma4 and instructions. I'll work on making a version of all of this that is standalone and not specific to DGX Spark.