Comment by reidrac
3 hours ago
> on a decent speed
But you said 7-9 tokens/second, that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.
3 hours ago
> on a decent speed
But you said 7-9 tokens/second, that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.
I think that any workflow that requires the user to stare at the tokens being generated live is using it wrong. Delegate, don't stare!
https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text
You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of like using a 3D printer: It doesn't matter if a print takes 10 hours, because when you come back in the morning it will be done.
Yes, with top-tier GPU farms you can hit hundreds of tokens per second. But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
Once you've used a model that runs at hundreds of TPS, it's hard to go back. Everything completes so quickly that you can iterate without breaking out of flow state. My biggest gripe with slow (<50tps) LLMs is that I've lost all the mental context I built up by the time it's done, which makes it extremely difficult to explore or iterate on solutions.
Filament snaps at 1am and then you have to run print again. 10 hours turn into many days potentially.
I watch tokens to see if it goes in right direction. If model goes off the rails, then it is time to stop and adjust prompt.
We clearly have different goals. I want an LLM to review my code, not the other way around.
It's still the same thing, you can ask it to do a full on report give explanation and details be thorough and then go do something else, another task a lunch break whatever and it will be done when you're back
3 replies →
> Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete.
If I spend 10 minutes reading an article, that would only generate 3000 tokens.
That’s not counting the prompt processing time.
We have very different expectations for LLMs if your tasks only take a couple thousand tokens and you’re happy waiting 10 minutes for it.
> Yes, with top-tier GPU farms you can hit hundreds of tokens per second
My 5090 gets hundreds of tokens per second with this model. No farm needed. I’d have to double check but I think even a $1000 Intel B70 might break 100 tokens per second.
> But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
If that old Xeon pulls 200W from the wall and you pay national average electricity costs, it’s going to cost $0.90 per day to run it.
I would rather pay a dollar per day, get my answers 100X faster, and not have an old Xeon heating up my house.
You could suspend it to ram, and only wake it up on request, it takes 2 seconds on my box.
1 reply →
We aren’t there yet. Not for frontier development work at least.
Except often queued agentic flows must be checked in on. Or to use the comparison, 3D printers are not immune to making spaghetti all night when something goes wrong. (I’m not a 3d printing expert so maybe that is solved now)
It is common for agents to just stop because overload or some API error hijinks.
Or you get a TUI question that is blocking.
In general you’re right though, staring at tokens from agentic is not time well spent.
Some of these I’ve built custom harness around in iterm2 though.
is there a good tool to manage these workloads? batch process a bunch, handle failures, retry things etc?
[dead]
For me, at least for agentic use, you need at least ~40tps. Less might be good only for tasks you could run in the background (like at night maybe).
Instead of coding agents like tool for local models I would like to see more "docker ps" like tools where you can queue up tasks that get processes incrementally (maybe when the pc i idle) and are specialized for doing retries and caching as much work as possible to work decently with slow local models.
i am working on making it faster but to me 7-9 tokens/sec feels very good. it was 0 tokens/sec a year ago.
Ignore the haters. What you've done is incredible!
If you’re interested in these projects you should check out the project this was based on: https://github.com/JustVugg/colibri
It says so right in the readme. They’re not hiding anything.
Yep, and we don't even know how long they spent on prefill. A typical 50-100k token session could take 10-20 minutes to prefill on a Mac.
Pending any new hardware or radically different LLM architectures, we're going to be waiting a lot longer than 2027 for 200B models on local hardware. SOC platforms like Apple Silicon are hamstrung by their obsession over raster performance, the hardware lacks the fundamental GPGPU hardware to be a replacement for real-world inference.