Comment by Kirby64
10 days ago
It’s not just old, it’s also tiny and quantized. It’s llama 3.1 8b at 3/6-bit quant. This is the type of thing you can run on almost any device…
10 days ago
It’s not just old, it’s also tiny and quantized. It’s llama 3.1 8b at 3/6-bit quant. This is the type of thing you can run on almost any device…
I get that, but not at 15k tokens/s.
But it’s irrelevant. 750 tokens/s on a full frontier model is useful. 15000 poor quality tokens is much less useful no matter how much scaffolding you put around it.
You are missing the point. This is a technology demonstration on prototype hardware, and no one intends it to be seriously useful.
Their architecture has fundamental speed and efficiency advantages over GPUs or Cerebras. They expect to scale up to real LLMs by splitting a model layer-wise across several chips, which they can do without incurring any throughput penalty.
3 replies →
I’ve been using 1,000 t/s on a near frontier model for a month now. It’s very useful for agentic coding.
It does require new approaches for me personally since I get a lot less time to think or read its output.
2 replies →
I think you missed the point and don't understand / aren't considerate of SLM utility.
6 replies →
[dead]