Comment by walrus01
2 days ago
My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours.
0.05 to 0.1 tok/s on the other hand, as reported in the URL for the lowest class of hardware, isn't really usable for much.
edit: I think this is a fantastic project in general concept, and look forward to seeing more efforts towards the general idea of being able to run a 350B to 900B size model locally, even if as slow as 1 tok/s, on hardware that ordinary people can afford. Anything along the general concept of "we have fast read NVME SSD storage, we have a big ass model on local disk, we'll read it at 11GB/tok as we need it, not try to load the whole thing".
The funny thing is Claude Cowork has taught me to be patient with response timelines. I’m now figuring I’ll be running locally no later than 2028.
(I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)
For 10k you can buy a used dual socket Intel or amd based rackmount server with a terabyte of ram, and run models on cpu only at a reasonable speed. Same server would have been 4-5k a couple years ago before ram price rise.
Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add.
Which CPU gen are you suggesting, is there any writeup on such setup where <10K (not incl. power bill) cpu only rig is giving usable token speeds on latest SoTA open weights models?
In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment.
Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control.
Kudos to OP though, They had clear goals and they achieved it.
16 replies →
This was the case a few years ago, but now the RAM costs twice that much by itself, and the server is sold without RAM.
> (I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)
The question is, will you want to run a model comparable to today's (meaning 2026) SOTA in 2028? Humans always want the latest shiny LLM model.
> I want to run a model comparable to today’s SOTA.
It says so in the quoted text, yes.
Today's SOTA also sounds totally sufficient to me, but I wonder how much our standards will inflate by 2028. Maybe a lot, maybe not at all...very hard to say.
This seems to vary by person. I get immense value in coding assistance from Qwen 3.6 35B-A3B which is like a frontier model from a year ago. But a lot of people say it’s stupid, useless, a toy, etc. I do work by the “short leash” method and mainly just use the model for brainstorming/planning/design assistance and zipping through the drudgery of boilerplate and executing refactors. I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway.
9 replies →
Looking at how critical we are about today’s models, vs where we were last year, and I don’t expect anyone to be content with Fable-class models in 2028.
Expectations seem to be rising at a faster rate than models can improve.
I’ve been wondering if chat is the wrong interface for slower local models (and some projects) and maybe something like a ticket system is a better fit. I just decided how I would test this idea on my available hardware before I go drop money on a Mac Studio or GPUs. I’ll probably have a POC this week. There is nothing novel here, just need to spend the time to get it working for me.
Having a thin python/ts orchestrator and workers that pick up tasks from the directories like events and decide whether to make deterministic calls and wait is pretty standard albeit custom way of doing things in this space where you're bottlenecked by the concurrent call your workers/agents can make.
The hard thing is always keeping complexity low and being ZeroOps.
Are there any frameworks/scaffolding/harnesses or general resources on this you can share? I’d love to learn more
2 replies →
So I’ve been thinking about this problem a lot, specifically as it relates to running LLMs at home, and I’ve been using GLM-5.2 to make an SMTP/IMAP-to-LLM gateway.
https://tangled.org/clee.sh/posthorn
I’ve been wondering about something similar - a system that enforces (or does the heavy lifting) of dividing a large task into smaller sub-tasks so that it’s easy to run/check/test each one independently - even on a fresh model instance if needed.
This is based on the observation that the medium-sized open weight models (~20-35b) are very able to one-shot smaller discrete tasks but seem to lose their way project managing themselves through larger tasks that have multiple steps.
This is actually really smart. It would be like working with a team of humans.
I have a 3 Mac Studio set up and built an IDE / harness (propelcode.app) and would be interested in contributing if you’re open to collaboration
Time to make EmailGPT
Use the ticket system built into mininote.ink 's mcp server. Works perfect right out of the box. Also great notetaking app.
Docs:
https://mininote.ink/docs/mcp-docs
Now many mini-PCs and desktops are able to read simultaneously from 1 PCIe 5.0 SSD and 1 PCIe 4.0 SSD. This can ensure a reading throughput around 20 GB/s, i.e. 20 times faster than on author's system.
With only 1 PCIe 5.0 SSD, the reading throughput is still significantly more than 10 times faster than on author's system.
So it is likely that inference speeds around 1 token/s are achievable on something like a NUC mini-PC.
I'm curious, where are you seeing M.2 2230 to M.2 2280 size NVME SSE that exceed 4.5 to 5GB/s sequential reads for large files such as a GGUF (likefrom an ordinary ext4fs file system with default options)? The PCI-E 4.0 or 5.0 bus they're attached to might be capable of greater speeds, but the bottleneck is the flash and the flash controller.
The benchmarks I'm seeing for many of them don't really make me think that a pair of consumer grade NVME SSD you could fit in a mini-PC or mini-itx size desktop would, added together, be capable of 20GB/s reads.
I do not think that I have ever seen any benchmark for a PCIe 5.0 SSD that did not have sequential read speeds well over 10 GB/s.
If there were such a slow SSD, it would not make sense to buy it instead of a cheaper PCIe 4.0 SSD.
For PCIe 4.0 SSDs, I have seen a very large number of benchmarks where the SSDs achieved read speeds close to the theoretical limit, i.e. around 7 GB/s.
Searching now randomly for recent SSD reviews, I find many reviews for "SanDisk WD_BLACK SN8100", which achieve between 13 GB/s and 15 GB/s reading speed, which is better than most other consumer PCIe 5.0 SSDs.
Of course, if you write a very simple program that invokes something like "fread" or "read" in a loop, you will not reach such speeds. Achieving a SSD throughput close to the limit requires a more complex program that can ensure that the SSD controller is permanently busy with pipelined read commands.
In the readme you can see benchmark which everyone with different hardware is running Colibrì, and I have to say I've seen great times! I'm always doing more to improve!
I have a 16-core system with 256GB RAM here I could try it with but regretfully it's so old the CPUs aren't AVX2 capable. Otherwise it makes a fairly good llama-server test system for CPU only stuff. Oh well. Time to upgrade (painful to the wallet these days).
Maybe we can see some integration!
3 replies →
0.05 to 0.1 per sec could still be quite useful if it was the speed for inferring a whole batch of tokens concurrently. Of course this actually requires fairly good SSD read performance (since you need to read a sizeable fraction of the complete model at every token batch in order to get good reuse) and is ultimately limited by CPU/GPU thermals which are a tight constraint on typical inference platforms. It's also only really feasible with tiny KV caches, which requires either a very small context or sticking to KV-cache efficient models such as the DeepSeek V4 series. Still, this might be one way of making use of existing lower-end hardware for practical inference of non-tiny models.
For most projects the more practical solution is to use clouds offering GLM 5.2 for free. 1 token per minute is minuscule compared to their rate limits for free usage.
But it's about the journey not the destination. My current running local LLMs train of thought...
And it's also about privacy. I just can't wrap my head around the fact people completely ignore that aspect when they compare on-premise and cloud solutions.
> on hardware that ordinary people can afford
These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd?
sigh
> These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd?
You can, right now, buy a brand new Mini-PC at or above this spec for $600 at retail [1]
Of course, if you want it in a desktop format with a much faster CPU, its going to cost you more.
[1]: https://www.amazon.com/GMKtec-M6-Ultra-Upgraded-Computers/dp...
The very boring pair of two 16GB ddr5 6000 I had in my newegg shopping cart went from $399 to $475, so increasingly the answer will be "no".
I bought whole Intel N100 mini pc with 16GB of DDR5 in it in 2023 for $AUD289 (so about $US200). I got a 16GB (DDR4) SODIMM in 2022 for $AUD88 ($US60).
Does it have to be DDR5? Is the limit RAM speed, or SSD speed?
1 reply →
Maybe that's a measure of the self-fulfilling dollar incentive toward "renting" someone else's RAM in the future rather than trying to actually own such an outlandishly luxury item :\
After 18y of thinkpads, this year I bouth a Lenovo yoga for... Cheap (1000€).
32G RAM, nvme 1TB, core ultra 258V.
Looking at the prices now... Wow, was I lucky.
Tried some of the 7b models locally, more than usable, around 30token/sec, not with the NPU, but using the ARC integrated GPU.
I am a noob for this, but I guess it's time to experiment more with this local setup
Maybe not afford new, but they probably already had it from before the current crisis?
Ideally this engineer's approach will yield better performance on lesser equipment in the future, if they keep up the good work after they get more-capable gear to experiment with as time goes by :)
[flagged]