Comment by kolinko
1 day ago
60 toks/sec may be ok for pure chat or token generation, but not for token preprocessing nor any sort of agentic work.
1 day ago
60 toks/sec may be ok for pure chat or token generation, but not for token preprocessing nor any sort of agentic work.
>.. nor any sort of agentic work.
may I ask why?
oai luna runs at 60-70 toks/s; astra runs around 35-45 , are you just worried that your agentic work can't spread across a lot of parallel work?
The numbers you povided are for output/generation. Input is around 5-20 thousand for Astra, and 16-60k for Luna. So a few hundred times faster.
Most of the tokens consumed by agents is reading stuff - docs, tool outputs, code etc. Previewing a content of a dumb single page of html is a few thousand tokens.
If you use claude or some other harness you can ask it how many cache writes/input tokens are there in avg session, and how much time it would take if preprocessing speed was just 60 tok/s.
Latency is the remaining hurdle for local AI. We'll get there, but my prediction is that it will take around 10 years.
I would guess the assumption is that "agentic work" should be happening asynchronously in the background on a machine that's also used for foreground interactive work, where the slower the inference is, the longer the machine's GPU and [V]RAM are "hogged" by it at a time, and so the more the machine's performance seems to degrade.
My self hosted LLM runs at 60-ish tok/s and does great at agentic tasks. I think that’s about the threshold where it stops feeling annoyingly slow.
60tok/s prefill or tok gen? If you use gpu and keep it in vram then I you most likely mean token generation, nit prefill.
I have a couple of V100 and am working on making Flash Next really fast on them. So far I am at like 193 t/s decode on a specialist 3-bit quant. On shorter prompts. Around 109 t/s on a ggml/llama fork at depth before I gave up and started writing the engine from scratch. I guess I will end at about 220-250 t/s at 128k context size with about 3K prefill on these cards.
https://stealthgoat.com/deepswe-flash-next.html (Total AI pile). I benched this quant for 3 days on DeepSWE in a 16GB VRAM gaming PC. It held 90 t/s on avg I think. It’s solid. The authors claims of minimal intelligence loss hold. Sorry it renders bad in mobile will fix it soon.
It feels great to have a local model ripping at these speeds. I now use Flash NeXT as an active agent managing part of the engine build process and let it do some coding on easier chores :)