Comment by mickeyp
8 hours ago
If all you're going to do is chat with an AI, you don't need a 400B parameter MoE model.
13 TPS is unusable for agentic coding, because if decode is that slow, then prefill is absolutely abysmal, and agentic work live and die by prefill speeds.
I’ve been agentic coding at home with like 15 TPS on average. I don’t mind it at all. I can give my agent a big task or a list of little ones and just… do something else. Why do I need to watch it work?
Seconding this. 2-bit quant of DSv4Flash 0731 (Unsloth) running on lemonade-llamacpp on a Corsair Strix Halo. OpenCode had an issue giving me 400s constantly for the past few weeks, so I just went all-in on local dev with omp. Yes, it takes time, but also: it works quite well. I've since gotten OpenCode to behave, so I'm back to using that quite a bit, but the local machine is really good as a "smol" model in omp, for example.
> If all you're going to do is chat with an AI, you don't need a 400B parameter MoE model.
Why wouldn't I want the smartest possible model I can get? I only "chat" about smart person stuff.
Why does agentic work live and die by prefill speeds?
It doesn’t. Decode is by far the most dominant and expensive part of agentic tasks and once prefill hits a certain achievable speed decode dominates the time in ant agentic flows with reasoning. There is a reason input is so much cheaper :)
Isn’t input so much cheaper because it’s just less work? Prefill is usually 10-20x as fast as generation.
If you have 200k context, but only 200t/s prefill, then you’ll wait multiple minutes when continuing an average session from yesterday. Or compaction - might take 15 minutes.
2 replies →