Comment by Aurornis
9 hours ago
> 150k TPM limit on public endpoint means that it's likely unusable for many coding tasks.
I don't understand. How does that make it unusable? Is the limit shared by an entire team at once?
150,000 tokens per minute is a lot. You could start hitting that with a lot of concurrent requests in your session, but even throttled to 150k TPM it's still going to be faster than anything else you find.
I think the 128K context limit is the real ceiling. These models aren't amazing at long context, but once you account for a short input prompt, the input files, and headroom for a compaction summary, there isn't a lot left for the problem.
It's a limit on input tokens. So that's 3 50k requests per minute. At Cerebras speeds, that's about 5 seconds of usage per minute.
I was very excited last year for their coding plan but seeing a burst of requests pulse and then sitting there watching the cooldown reset is really not a great time.
Even though each individual request was fast, the sessions were only maybe 10% faster on wall clock time since there was so much waiting time.
They made the coding plan a bit better toward the end, but it was pretty tough to use throughout.
Seems like an Amdahl’s law of inference economics? there’s so much compute relative to SRAM on the chip and shoreline bandwidth onto the chip that caching buys ~nothing? The contended resource is SRAM and a given token of context needs just as much as another.
That’s not what caching is for. Caching lets you resume with a pre computed KV cache saving you from having to ingest everything in the chat history as input on every single round trip. You still need caching regardless of SRAM or not as it saves a huge amount (and ever growing) of compute ingesting the preceding history every time you want a completion.
I don’t know why they don’t give a price discount. Maybe their hardware is incapable for some reason of saving/restoring the state? Or maybe they just haven’t built the infrastructure to do it?
Can't you do something with multiple accounts?
You would lose caching (if they cache)
Or just buy a 5060. This will run on most any 16gb card. Slower for sure but far cheaper than another subscription.
2 replies →
128k context is not a limit of the model, that's a limit of implementation:
"Context Length: 262,144 natively and extensible up to 1,000,000 tokens."
https://huggingface.co/Qwen/Qwen3.8-27B
TPM means Tokens per Minute.
GP is referring to GGP’s last paragraph. 150k t/m, yes, and 128k context.
150k tokens per minute at 1.5k tokens per second means you can have like 3 users concurrently and that's not a lot.
150k by account. At 1.5k a second you hit it very quickly.
Exactly, it burns the tokens 3000x faster, which means the budget ($$$$$$) runs out so faster it will stop super quick, not able to perform long-duration work. At 27B parameter size, the intelligence is not able to accomplish work within a short amount time. Consequently, it become not usable.
I (we) run Qwen3.8-27B-FP8 on a DGX Spark box - that's roughly £4000 of hardware.
I did benchmark it in various ways and it runs quite well but it is a quantised jobbie and 1.5k t/s is also rather faster than anything I can possibly hope to achieve.
To run that model at those sorts of speeds is going to need some serious investment and you are going to have to pay for it.
4 replies →
Unusable is too stretch IMO, you can still use it in tiny tasks that’ll respond almost instantly