Comment by windexh8er
10 days ago
What about 15k tokens per second? [0] I remember looking at this earlier in the year and it being so fast that it feels fake. And, yes, this model is old - but still awesome for what it is.
10 days ago
What about 15k tokens per second? [0] I remember looking at this earlier in the year and it being so fast that it feels fake. And, yes, this model is old - but still awesome for what it is.
It’s not just old, it’s also tiny and quantized. It’s llama 3.1 8b at 3/6-bit quant. This is the type of thing you can run on almost any device…
I get that, but not at 15k tokens/s.
But it’s irrelevant. 750 tokens/s on a full frontier model is useful. 15000 poor quality tokens is much less useful no matter how much scaffolding you put around it.
15 replies →
I just tried it, and the answer is non-sense.
I asked it something simple, list some good indie puzzle games, and half the answers are games that don't exist. Imo quality > speed.
They baked the LLM into a CPU
at 15K tokens/s... do you need code anymore
Yeah, that's the point, right? With tool calling the LLM becomes code. So instead of asking it to write an accounting software, you can hire the LLM to be your accountant.
But you'd still need code if you need something done in a consistent way.
5 replies →