Comment by gandreani
10 days ago
Easily the most interesting part of this announcement is buried in the second to last paragraph:
"We're also launching GPT‑5.6 Sol on Cerebras at up to 750 tokens per second in July, bringing frontier intelligence to customers at unprecedented speed. Access will initially be limited to select customers as we expand capacity."
750 tokens/s on a frontier model is going to be extremely interesting. I doubt this new version is anything but a version bump in terms of capabilities but if we can start getting these answers back faster, they end up being more useful.
Just off the top of my head, I can think of the tedious task of finding certain functionality within a codebase. I usually can't beat an AI agent harness at this task today. If the AI model is 3x faster I have less of chance.
https://mikeveerman.github.io/tokenspeed/?rate=750&mode=thin...
This is what 750tps looks like, I guess.
You get used to it. I don't even see the code. All I see is blonde.. brunette.. redhead.
That’s an awful visualization. I can skim code quite quickly, but not when it shows up one character at a time in a small window, modem style.
At least that site should draw out a full page then start replacing that page with the next, starting from the top and working downwards, repeating each time it hits the bottom.
This is how tools like claude code and chat prompts output their tokens, so I'd say it's actually a pretty good visualisation.
1 reply →
That's exactly what it looks like in the tools I use most (opencode and codex), so for that purpose it's a pretty good visualization.
> I can skim code quite quickly
are you by any chance hyperlexic? interested to hear more about this, like how fast is considered fast
Just to think what this will look like in a couple of years.
Hopefully like this (but smarter): https://chatjimmy.ai/
36 replies →
I started with a 2400baud modem, I've seen how this goes
Sometimes I visualize a setup like this [0], based on 2D art by Simon Stålenhag. Someone has their home robot sitting on a desk connected to their old PC with thick cabling, dumping endless lines of each subsystem's <think> logs to diagnosis why it did something weird earlier in the day. Systems pushing 750+ tokens per second per subsystem might even be considered on the slow side for realtime tasks by then.
[0] https://www.therookies.co/entries/39513
Probably not. Everyone will still need a lot of reasoning tokens and tool calls. Running the tests for every round is tiring but must be done.
Imagine a Beowulf cluster of these…
6 replies →
Probably will not be looking at text like this in a few years.
probably something like this https://sb0xw.csb.app/
For comparison, openrouter says opus 4.8 is ~55 tokens/s and fast mode is ~102.
750 tokens/s for their largest model is going to be nuts
What about 15k tokens per second? [0] I remember looking at this earlier in the year and it being so fast that it feels fake. And, yes, this model is old - but still awesome for what it is.
[0] https://chatjimmy.ai/
It’s not just old, it’s also tiny and quantized. It’s llama 3.1 8b at 3/6-bit quant. This is the type of thing you can run on almost any device…
17 replies →
I just tried it, and the answer is non-sense.
I asked it something simple, list some good indie puzzle games, and half the answers are games that don't exist. Imo quality > speed.
They baked the LLM into a CPU
at 15K tokens/s... do you need code anymore
7 replies →
Using gpt-5.4-mini in off-peak hours already feels like super-speed to me. That's probably no more than 100-150 tk/s. I can't imagine 750!
I've always eyed Cerebras but never had a use for it that would justify paying for the API directly. Although now that I think about it, trying out the API would probably cost less than a subscription for a month...
Try gpt-5.3-codex-spark - it's 1000 TPS and from my experience more capable than 5.4 mini.
If you have a subscription it's a different pool of usage.
7 replies →
The ChatGPT subscription gives you access to the -spark model(s) in Codex which are blazing fast (but pretty dumb) which I think runs on Cerebras hardware too.
4 replies →
I have a pretty good use case for gpt-oss. The amount of time savings has actually been wild. Definitely worth a try. Just to be clear, it gets like 2000tok/s
But it seems that there is some queuing/load balancing on their side, I mean when opus is actually outputting this 55t/s it feles fast, but apart from it's internal reasoning I think there's sometimes just waiting.
Oh wait yeah good point. At 750 tokens a second and the same amount of human patients they can set it to think for the same amount of time but four or five times the amount of thinking tokens, which may improve the quality of the eventual output.
"up to 750 tokens per second"
Emphasis on "up to". Imagine whatever limited situation (e.g. pre-cached query) and that will probably be the only time it hits 750 tokens/sec.
the more advanced models also utilize a lot more tokens, and a lot of these extra tokens may go towards safeguards at a higher rate than prior models as well.
not to say a speed boost isnt there but if they didnt increase tokens / s at all youd likely see things slow down a lot with the new model compared to current
I think regular users will still have the old speed, so should be easy to tell whether it is more thinkier than 5.5.
> I can think of the tedious task of finding certain functionality within a codebase. I usually can't beat an AI agent harness at this task today.
Yup, I remember "racing" the AIs to figure things out in codebases just a year ago. Today, I have no chance. Whether it is due to degraded reasoning capabilities on my part or better models, I don't know.
At least in my case, much of the code in the codebase I'm working on is AI generated so even if I have an accurate mental model of how everything works, I have no idea where any of it is located or named.
To be fair, whenever I join a pre-existing code-base [1], it's the same. I have no idea and have to map it out ;)
[1] Not AI codebases (and of course, AI code bases I guess)
4 replies →
AI is always going to be able to write a grep statement faster and more accurately than a human
When AI is ready, it won’t need to grep at all. That is, it will train on the data in-situ instead.
Now start thinking, if possible.
https://www.youtube.com/watch?v=43QHhEfzz-Q
I'm skeptical of how fast "up to" 750t/s really means. Maybe if they make it extremely expensive so it frees up enough capacity?
GPT‑5.3‑Codex‑Spark currently runs on Cerebras chips and it's giving me around 150t/s. Still relatively very fast, but nowhere near the 1,000t/s they claimed at launch. (Also it's not a very good model.)
That said, I'm super bought in to faster models being better for most use cases than smarter models.
If it's 150 t/s, that's barely faster than Nvidia GPUs who are batching a lot more and are a lot more cost effective. Add in the Groq piece and Nvidia claims it can do 400 tokens/s.
I assume it’s just oversubscribed. I’m sure it “can” go faster. But yeah that was my point.
Soon the bottleneck will be how fast your laptop can grep for a string.
At a certain rate we will be able to move towards continuous / real-time inference systems. The discrete, turn based solutions are quite confining with how they must be trained. Continuous and real-time would fundamentally alter the domain.
From an information theory perspective we are still in dial-up territory with regard to the actual information rate. 750 tokens per second would be a really bad dialup connection. Imagine 10 millions tokens per second.
We still have the problem that auto regressive decoders are memory bound.
The new Blackwell hardware combined with TensorRT-LLM and speculative decoding consistently can hit 1,000 TPS/user barrier, comparing to closer to ~250 TPS/user (out of 10k+/TPS on the server)
Is there something I missed, this looks more like 14.4 to 56 on a 64kbps backing channel modem story. I have no doubt that there are still massive gains to be found, but they seem to be using existing constraints more efficiently, not that fios is coming.
I don’t have the budget to work on the foundational model scale, but with a draft model 10x–20x faster than target and an 60-80 acceptance rate I can see how they could promise 750/TPS (with a lot of other hard work) but I would appreciate where I should look to figure out what I am missing.
agree, from my POV the constraints are still there but we've optimized now. still haven't solved the core problems.
1000TPS - what model size?
1 reply →
Is there anyone exploring or writing about this in public? I've felt for a while that the turn-based model was not quite right, but also felt too stupid and ill-informed to have much of an opinion about what else it could be.
I have an active 'sleep' mode, where when the user is AFK the LLM goes into a loop with a sleep 10 between turns, and determines (via tool use) if something should be done. That's still a 'turn' in a way, but it's all the LLM just sort of sitting around like a human would, pondering what to do next.
But I could imagine after each space(eg, word) having a 27b model on a nice rig, with thinking off, doing a quick look at the sentence and determine if it should interrupt and start a real turn with thinking on. Which kind of is non-turn based in a way. If you're typing fast, it might hit that run every 3 or 4 words, but that's sort of how a human might be when a person is talking to them. That is, waiting for enough info to interrupt, if needed.
There might be a way to process chunks of a sentence using commas as break points, eg for comma delimitated phrases in sentences, so the whole sentence doesn't need to be re-processed each "should I break in" assessment at word break.
Could be fascinating. Could actually do some of this right now.
I don't think this is what the parent poster was thinking, but the idea even at this level seems fun.
3 replies →
Thinking Machines, the started founded by former OpenAI CTO Mira Murati. The interaction models demo’s in their videos imo breaks the awkward turn-based barrier. Returning responses quickly reaches a threshold where it starts to feel like a natural conversation. Their approach to solving this problem is rather clever.
Your comment made me think of another real time. Real time, dynamic code/apis.
Imagine a world where there is no code, just things mildly handshaking and then creating data APIs on the fly. Where communication is fuzzy and locked in on an individual basis. No years of RFCs, no RFCs at all, just... data.
Just data, man.
An API arbitration aberratically assigned at authorized access, abridged and annotated, analytically assuring absolute assurance.
Why remove the code and binary artifacts, though? Don't you want to verify that the business logic is accurate and the processing is deterministic?
In some circumstances there is no substitute for something that you know will produce the same answer for a given input, consistently. And that's before even considering the watts per response.
3 replies →
It's very easy to see how world changing this technology will be. In a few years these AIs are going to be negotiating how they communicate with each other. Humans won't necessarily be included in that negotiation unless we have some kind of specific reason to. So many communication layers are going to be opaque to humans. We just have to trust our AIs are communicating efficiently and safely.
1 reply →
I'm pretty sure the LLM will get fed up and start writing an RPC
Also > An API arbitration aberratically assigned at authorized access, abridged and annotated, analytically assuring absolute assurance
Cool that you wrote all the words starting with "a" but I don't understand what you mean
What this made me think of is life before computers, where people mildly handshake, create agreements on the fly. "Where communication is fuzzy and locked in on an individual basis."
TBH, to me, this imagined future looks a lot like it'd have all the problems we already have.
I made this https://github.com/alehlopeh/hallu
1 reply →
Wow. Sci-fi stuff!
I’ve thought about this before. No flaky config files, no updating endpoints, no status monitors. Just fuzzy everything that works almost all of the time.
And more importantly those 10 million tokens/s should cost fractions of a penny. Tokens need to be dirt cheap so I hope they build out massive solar+battery powered data centers asap.
No anything but wasteful, weak, expensive, environmentally harmful solar. Nuclear is the only path forward for superior energy production, at least until we figure out fusion.
1 reply →
That would be interesting.
Do you feel most of the speed upgrade will come from the software or hardware side?
Ahh yes slop at the speed of light, how useful!
AI is improving and seems to be reaching the point of not being slop (I am talking about flagship models).
3 replies →
Yep this is a glimpse into the future of 500+ t/s, which is in my opinion the next big thing that validates Jevon's paradox (the models are already smart enough)
“Smart enough” really depends on how many other people have encountered a problem close enough to yours and solved it somewhere on the open internet, IMO.
Most of the frontier models can, when prompted and tooled correctly, do a lot of “reasoning” tasks that amount to resolving how the user has explained a particular widely known paradigm.
The more difficult and obscure the issues you provide them with, the faster you notice them reward hacking by altering the criteria until they are no longer attempting to solve the problem. Using “advisor” style loops helps hold this off at the cost of tokens, but there is still a fairly short limit at which they will essentially give up if they can’t find all of the necessary information - sometimes the issue is actually worse if they find a small amount of information instead of nothing - they’ll extrapolate from that tiny piece of data and generate plausible-sounding hallucinations almost every time.
And god forbid your problem involves doing something a different way than the majority of people do it. Unless you can write a full spec on it, the models will repeatedly spiral back into adjusting everything about your problem until it matches one of the most popular approaches in their training data.
> how many other people have encountered a problem close enough to yours and solved it somewhere on the open internet
I'm 100% sure that all our web, cc, codex or whatsoever sessions are used in the training, RL or either both.
This makes the size of the universe models know about at least one order of magnitude bigger than the open internet.
6 replies →
This may have been the case one year ago, but with contemporary models such as Opus, I run into this less often.
1 reply →
Faster tokens = more reasoning loops, so it can actually make the models smarter as well.
Yeah! So at a much smaller scale, being able to boost Step 3.7 Flash up to 40tk/s on my Spark-alike with proper triple head MTP was the thing that made it superior to Qwen 3.6 27B in wall clock time despite Step reasoning more
A lot of the open Chinese models get their results through huge reasoning loops. Being able to boost decode perf is what will make them worth it, and I’m sure OpenAI and Anthropic could do similar (if they aren’t already)
[dead]
I think the glimpse that is there will be exclusive access. So much for the open in openAI. If this technology really transforms society in the ways expected with inequality an unavoidable consequence equal access should be required like internet access was (isp can’t give preference to specific user traffic)
[dead]
I saw videos of coding with Mimo-V2.5-Pro UltraSpeed, which is advertised at 1,000 tokens/s, which is very impressive.:
https://www.bilibili.com/video/BV1fME16uEW7
If the time-to-first-token latency also greatly improved, this could be very useful for end-to-end in controls, like autonomous driving for example.
It’s awesome, particularly since it’s at DeepSeek tier prices (3X of DS-V4-Pro). At 1,000 tok/sec though you can really rip through tokens. (About $9 an hour if you manage to run the output nonstop.)
It tends to cost more than DS since it doesn’t seem to have as many input cache hits.
bean in mind that "GPT‑5.6 Sol on Cerebras at up to 750 tokens per second" not necessarily means the same model (in terms of inference result). It can mean anything like a very quantized model, a different level of model activation per inference etc.
Of course we can trust that wouldn't name the same thing with different levels of intelligence, right? Right?
yeah but it’s trivial to just try it out and compare.
"we can start getting these answers back faster, they end up being more useful."
Dude, 10x token speed is going to be absolutely nuts. Half the "parallel subagent workflow" business seems to be driven simply as a means to avoid tapping your thumbs waiting for the infernal robot to finish something. If things come back speedy quick all the time, it should keep up with the "speed of the human" and let me stay focused on one thread instead of half a dozen. Plus the cost of screwing up gets significantly lower because you just re-fire with an adjusted prompt and iterate.
Someday these things will be 100x as fast as they are today and that is when things will get insane.
it also makes the parent brain-dead because all those subtokens are missing from the context thus unable to steer the hyper dimensional context driven generation, and the subagent is dumb as a post so synthesizes something very weedsy while you're specifically attempting to understand the forest
You have an agent spawn the agents for you! You can ask Claude to do it for you, he is happy to use sonnet when you ask for grok and opus high when you ask for deepseek.
I still use GPT-5.3-codex-spark which also runs on the Cerebras chips. Spark can run at >1000 tok/s but it's highly limited in it's context window size so it's not suitable many workflows.
Granted this will be a bit slower (relatively speaking) but it will still be awesome.
Same - I had some "AI-assisted coding interviews" where I had to bring my own AI tools, and found the speed of codex-spark to be important for making progress quickly (and not sitting there waiting on Opus to think for 10 minutes).
Does the Cerebras variant offer input caching and corresponding discounts? Last I checked Cerebras would not cache or would cache but not give discounts for the cached input, making it impractical for agentic use and multiturn conversations.
> second to last
There's a word for this that you should never pass up an opportunity to use: penultimate. (You should also never pass up the opportunity to use "defenestrate," but it sadly does not apply here.)
A friend of mine had his visa accepted because of this. He was explaining what he plans to do in US and he threw in “penultimate” into a sentence somewhere.
The council stopped him, said that if he knows such words he definitely won’t overstay his visit to work as a dishwasher, and accepted his B1/B2. Seriously.
Not sure if it would be the same if he used “defenestrate” when talking about his plans.
>> penultimate
Oh that's a word I haven't heard in a while. And I know it mainly thanks to Monty Python great sketch:
https://m.youtube.com/watch?v=l9Aj7W3g1qo
Cerebras is Milli Vanilli. They spend 10 years burning cash on a failed idea (which is frankly insane, since they should have figured out the limitations of heir stack in like... a weekend) and struck accidental gold with their 'Giant ass wafer'.
The company is valued like they broke open the grail, when in reality it's more like they bought a Cybertruck, got it stuck in the mud, and realized "You know what this thing does better than all other cars... shovel mud"
I'm shorting Cerebras with margin to virtually zero.
That statement gives a lot of insights into the possible model size. Llama 3.1 405b runs @ ~900t/s: https://www.cerebras.ai/blog/llama-405b-inference
From initial vibe research (which is totally not correct by any means) ~13.5k concurrent streaming clients capacity
> I usually can't beat an AI agent harness at this task today. If the AI model is 3x faster I have less of chance.
Yes: we have these new tools that are extremely good at helping us search through our codebases. Not just to find where/how functionalities are implemented: IMO bug searching is even way more powerful.
But: why would you want to compete with AI to do that? I cannot compete with grep/ripgrep... And I'm cool with that.
This lets you focus more on the more interesting parts, where AI/LLMs suck fat balls.
This is a strange one. We know the hardware capabilities of Cerebras force them to do aggressive REAP pruning to serve Kimi K2.6. Meaning that about 750B parameters is the upper limit of what they can serve economically. Not sure if this means Sol is smaller than anyone thinks or that they're just going to charge so much that a very inefficient serving regime is feasible.
OpenAI also announced two days ago that they're starting to make Cerebras style chips themselves [0], will be interesting to see how fast SotA model inference will be by the end of the year.
[0]: https://openai.com/index/openai-broadcom-jalapeno-inference-...
I don't understand how you refer to this as "Cerebras-style". Cerebras is wafer-scale and unique. Jalapeno is an inference-optimized conventional chip.
I don't see any indications that OpenAI is doing wafer-scale work.
I tend to doubt they would. Cerebras notably doesn't have a kv, is wildly high bandwidth, but within/across the chip, not able to dump/restore kv super well. I doubt openai is going to build something that is as expensive to run. Also, wafer-scale is absurdly hard & weird to pull off, so I doubt that would be their first foray.
Cerebras is different than what jalapeno is.
Jalepeno is for mass scale inference.
Cerebras is extremely expensive and difficult to scale, hence the limited release.
Even if their chip is a difference maker, end of the year is wayy too optimistic. It’ll at minimum be a multi-year effort to bring it to production at scale.
This would be amazing for some of our "real-time" workflows, that need to fallback to AI for one reason or another. What used to happen is a rules based system did the majority of work, and occasional corner case would fall back to humans. Then we moved AI in, still not real time, but much faster. Cerebras could make that even faster.
It all depends on the context window size. A small context size with fast performance won't be very useful today, as most workloads (like requests behind codex) usually have very long context.
At thousands of tokens per second, LLMs (harnesses) can start to do a broader tree search of possibilities even in inefficient token space. This unlocks capabilities outside programming.
Last I heard, Cerebras chips were entire wafers and would be extremely expensive. How could OpenAI possibly have enough of these to serve a popular model at scale?
This is something Xioami already did with MiMo-2.5-Pro a month ago, and at a higher speed (1,000 t/s).
750 tps at GPT-5.5-Pro prices would be ruinous!
The speed sounds great，faster models make that gap much more visible..
3x faster burn than 3x expensive token, generate more tokens, more fees
this means they also earn at a faster rate in some setups :)
From what I know about batch processing/ concurrency in inference this is a pipe dream... Or its going to cost an arm and a leg. I think they're lying or its going to be a much smaller model and not "frontier"
You have speculative decoding that easily increases speed 2-4 times with no loss of quality, and of course MoA architectures that speed up inference 10 times or more, although with some quality loss.
Better hardware, and other techniques on top of that and you speed up even further.