Comment by bearjaws

17 hours ago

If you care about speed Cerebras gpt-oss-120b is 1400tk/s and "just as smart" in ranking.

I've used it on a few for fun projects and its decent but the speed is crazy to watch.

30 comments

bearjaws

Reply
ford  16 hours ago

Also kimi 2.6 at 1000tps (as of may), though when we reached out they had a >12 month waitlist and minimum 7-8 figure annual token spend.

[0] https://www.cerebras.ai/blog/cerebras-kimi-k2-Enterprise

  • walrus01  13 hours ago

    7-8 figures annual spend will buy a hell of a lot of capable local inference hardware you can own, though it won't be at the absurd token/s rate, you'll be able to run almost anything on it... And it'll still have a good residual resale value after 4 years the way things are going now.

    • podocarp  9 hours ago

      Feel like you could spend 6 figures building out a team and the rest renting compute for a whole year, and get the team to create a local inference solution with that kind of budget…

conception  13 hours ago

It was better when they had gemma at 1k. Inco does DS flash at about 600. A few places will do K3 and GLM in the hundreds.

Such a tiny model at that t/s is less impressive than it would have been four months ago.

  • physicallyIllfr  12 hours ago

    Lighting my codebase on fire at the speed of light. Like microwaving the spaghetti.

    I genuinly only see these speeds being useful for customer service/transactional workflows. Of which much smaller models can do the job (but those dont make tons of money for companies like Cerebras that need to pay off massive amounts of debt).

    Nobody needs to code at 600 words per second. Using a 100tps model for an hour or so will leave you with 4-8hrs of code review and revision work.

    • sbierwagen  10 hours ago

      Human code review? What is this, 2025? The modality today is write with one LLM, review by a different one, (important: two different model families will catch errors one series won't) then deploy right to production.

      4 replies →

    • calgoo  8 hours ago

      No you dont need to code at 600 w/s BUT at those speeds, you can start doing things like asking multiple different agents the same question and picking the best solution each time without noticing the lag.

      1 reply →

    • RugnirViking  8 hours ago

      > Nobody needs to code at 600 words per second.

      I do. I used to use haiku for the speed. Now its just as slow as the rest. Speed is my #1 ranking of how good a model is

      2 replies →

  • lostmsu  5 hours ago

    Inco sucks. I tried their GLM 5.3 Flash and it was quantized to the point of hallucinating Chinese in the middle of English only agentic sessions. Never happened with any other provider.

LoganDark  15 hours ago

Please do not try to use gpt-oss-120b over Cerebras. It is broken, screws up tool calls most of the time, forgets to end thinking blocks and has all sorts of other issues. The speed is amazing but it is absolutely not worth it, especially at that quite incredible cost. Think: $5–10/minute levels of cost with a single agent, because Cerebras also offers no cache pricing for input tokens at all.

  • bearjaws  4 hours ago

    Not been my experience, I have it using tool calls in a video game I am building and it correctly adheres ~99% of the time.

    I have it retry on failure, but you should do that with any LLM really.

    • LoganDark  4 hours ago

      I kept having experiences with gpt-oss-120b on Cerebras where it would get stuck in a thinking block and then start endlessly saying things like "Running the command now." or "Making the changes now." and then simply repeating similar sentences like that forever instead of actually making the tool call. It made tool calls other times, so it wasn't an issue with tool calls being impossible, but it just wasn't doing a good job of using them for real instead of simply saying it would. So this was not an issue of it starting a tool call and then putting invalid syntax inside of it, it just would not make the tool call it was supposed to whatsoever. There's no automatic way to retry that.

  • eli  14 hours ago

    Which is wild because it does, in fact, do caching

    • fakwandi_priv  10 hours ago

      > There is no additional fee for using prompt caching. Input tokens, whether served from the cache or processed fresh, are billed at the standard input token rate for the respective model.

      So what he’s saying is correct, there is no separate cache pricing, which by normal standards should be 10% of the cost, which can become exceedingly expensive for anything other than single turn. The way they are stating this is of course strange..

  • shard972  13 hours ago

    Yea i had some pretty meh results using gpt-oss-120b it in my evals where it should have benefited speed alot but it really under performed what i was expecting.