Comment by Mitchem
9 hours ago
While I am skeptical of the results here, I am very excited for this new trend of making models faster. Running capable models at 1k TPS is more valuable for me than running better models at 30 TPS. I can only imagine the trend continues to move from "let's only make models smarter" to just incremental intelligence gains but with step improvements in speed.
Why? I'm personally on the opposite end. Less babysitting/higher quality means more time goes back to me/the user. 1000tps of bad code means you have to keep validating the output and circling back.
High tps is good for deeper agent thinking loops and openclaw etc. I was running cerebus recently doing some data heavy tasks, it managed to crash the server I was submitting posts to. 6 hour task down to ~1hr
id rather iterate multiple times than wait 15 minutes to notice it made a mistake.
Again, my point is exactly the opposite. Higher quality implies a mistake isn't made in a significant % of cases.
3 replies →
So i agree with you, but there's no SOTA model that i don't have to babysit. I'm not going to just throw a large pile of code in there unreviewed, and so what i want is faster iteration on code in logical, reviewable chunks. Ie just like i'd normally write myself; small, logical commits.
Faster iteration means i mentally checkout less and am more involved with the code being created.
My hope is that in the far far future, we can get LLMs so fast that i can work in my IDE like normal and the LLM will just be an extension of autocomplete. I can state a goal, rough out functions, code, etc, and it'll just work around me like a very fast pair programmer / autocomplete.
The chat interface is an intermediate step that frankly i hate. The faster it is the less i wait.
Now for vibe-slop i'm making on the side, yea i don't care about speed. But that's not something i'm employed to do or anything i truly care about. It's a different workflow entirely.
I get it, you just prefer to do things differently
> Faster iteration means i mentally checkout less and am more involved with the code being created.
This is a good point I didn't consider and you're right. More interaction brings you closer to the code.
I still think that this is the opposite of what I personally want. Either I write the code (or a large majority of it), and be fully involved; or be more disconnected but more free to focus on other things. The middle ground removes me from the equation, but also requires me to babysit.
Indeed. For me opus 4.8 is good enough. If only it would be 100 times faster. You could run it in self verification loops much much faster. It sometimes takes 15 minutes for me to complete a simple task. For example configuring AWS agentcore and deploying an agent on it. Takes forever with Claude with constant issues it tries to solve.