Comment by K0balt
20 hours ago
These are going to be so awesome to buy in 5-8 years!
Right now I’m running 2x 3090 and 128 gb ram on 3.8 flash-next180b (the quant is 97gb) and it hums at120t/s. It’s really a surprisingly good model for general tasks, and I’m soon to see how it does with more agentic kit. For agentic work it does slow down quite a bit when the context fills up… but what do you want for running a 180b model on consumer hardware?
Right now I have it set up as an administrator for our asterisk PBX and radius server, and it does higher level judgment on our cctv footage for notifications to security. It’s been really good so far. Agents are the new UI, everything just works over SSH now.
> Agents are the new UI
Poorly-defined and ever-changing with occasional cases where a button does nothing, does something else, or deletes the whole prod database?
Idk. For a prototype they might be, but the whole reason we have this software thing going on is because we want some level of determinism.
> Poorly-defined and ever-changing with occasional cases where a button does nothing, does something else, or deletes the whole prod database?
This perfectly sums up all UI/UX decisions I've encountered in the ~ten years prior to AI, yes
Lol. I’m totally with you on agentic UX development. It’s frequently terrible, and you can tell which companies are telling everyone to 10x with AI. But the problem is testing, not the coding. My first stab at a UX render usually sucks too, it’s the iteration that brings perfection. It’s the same for agentic development.
What I meant though wasn’t agents building UX. I meant that in our case at least, the agent is the UX. “put a new extension 117 for Sam at the port next to Jane’s, and have it ring through to his cell when he’s not in the office. Leave the old 117 as is but only ring where Sam is, if that is known” basically, agents just do everything with SSH, and you can talk to them instead of poking around with a mouse. We don’t build ux for internal systems anymore. Local inference is strong enough to be reliable now with the correct tools and policies, and for 10-20k in hardware you are self contained. (Obviously $more if you are going to have a lot of people using it at the same time)
The “one wierd trick” is to build testing jugs for everything. Before a configuration is deployed, it’s tested against a toy instance, all existing functions are tested in all common cases, new functions verified, inputs fuzzed, adversarily abused, and stress tested against dummy loads. If its code, it gets examined against development policy, self containment, explicit interface walls, race conditions, code standards, complexity, factorability, and often A/B testing of different architectural choices. Except cryptography and a couple of other things, we generally prohibit the use of any libraries, it’s just not needed and the code never passes our tests anyway.
People are just YOLOing it and assuming that if it basically functions and looks right then it must be perfect. You’d have the same luck if you told your UX guy “test only the obvious, and if it looks about right at first glance, merge it. Don’t do extensive testing.”
The real thing with AI is that now you can afford to do a ridiculous amount of automated testing, setting up test fixtures, hardware simulators, mannequins of all the external interfaces, fuzzing, etc. Our testing code is 2-3x our product code, and we are shipping 10% of the bugs we shipped with humans, and it’s still 1/4 the cost and 4-10x the speed.
People shipping all the crap that is everywhere now have bad process engineering. Plain and simple. If you try to build software with AI like you used to with people, you’re going to have a bad time. The pipeline is very different.
> My first stab at a UX render usually sucks too, it’s the iteration that brings perfection
With AI, your iteration cycle gets 10x/100x quicker. For the same developer effort, you can effectively try out 10s of ideas the same period of time.
Iteration also has a human component. When a person works hard and writes the code, they are personally invested in its success. So iteration, or, chucking out someone’s hard work is psychologically damaging and causes human conflicts in teams. If AI wrote it, and AI deleted it, no-one gets upset. All that matters are good ideas. Reducing the human cost of iteration is just as important as the time saving.
And, now you have freed up implementation time, you now have way more time for testing, that thing that everyone wished they had more time to do, with a super intelligent 100x developer (ai) as your coding partner, and a 100x lever for automating regression testing. Bad UX has zero excuses.
1 reply →
kudos. Precisely this. People are doing agentic design backwards very often: YOLO then cry. But, if first you write skills, specs, set up orchestration, architecture and flows, including review layer, discrimination layer and trajectory management layer, then you task the agents with creation of verifiable testing and static analysis rig, based on the golden samples of high-quality manually engineered code, and only then you tell then to write conformant code.
Agents (and LLMs) are inherently non-deterministic. In coding practice, this translates to garbage code in the long run. The solution is simple - make the process as deterministic, verifiable and rigid as possible. Suddenly, you get all the bonuses of agentic flows (insane initial speed, trivial cost, express iterations etc.) with none of the usual problems... and suddenly, humans have hard time matching the quality of the final artifact with hand-written code, because we suck at testing, fuzzing and debugging - because we're basically single-threaded and overfocused. Agents, OTOH, excel at it, and context management is also trivial compared to humans.
This feels like a detached backpedaling pivot to save face tbh.
But anyway.
I'm sure there are people that use the tech wrong. That can be said about any tech in existence.
1 reply →
It works just fine. I've been using it for over three months myself once I tuned my MCP server. I run a 25-30 agent fleet with zero issues, I control it with Codex or Claude Code.
This isn't something you'd be able to manage with a normal UI since every agent has its own at of settings. My agents come in sets, I usually change a set or multiple sets, 10-15 settings per agent. And my harness hasn't made a single mistake once I worked out all the kinks from MCP.
"These are going to be so awesome to buy in 5-8 years!"
Can't wait. Anyone has a idea of what the price now would be?
The closest I can think of is AMDs MI350P it has 144gbs of hbm3.
The press says amds price is 30-40 thousand dollars. On eBay there is exactly 1 going for close to 90 which is crazy because for that amount you could buy a MI350X and all the related hardware to actually use it and it has double the vram.
So I would peg it at 40-80 thousand dollars given it's Intel, and it's lpddr5 not hbm and not gddr7 so performance is likely to be lackluster.
At work we've got a 96BG Blackwell card serving Qwen3.8:27b at well over 100tok/s. Most of the time it's way faster than codex. Just this morning I set qwen and codex on the same task and Qwen came back flagging problems Sol 6.1 missed.
Local models are going to absolutely eat OAI's lunch before too long. The only moat left is hardware cost, and they can't keep it artificially high for too much longer.
I use Claude Opus 5.5 and Codex Sol 6.1 in my job and I love it. I wish be able to run local models which have power at least current Opus 5.5 on medium hardware which does not cost millions $. This is my dream, running LLMs and AI agents to just create things for myself. On my really old hardware I can only run simple and small 1.5B model (Qwen2.5-Coder-1.5B) and well... Ihmo it works fine with offloading to GPU and using llama.cpp but it does a lot of mistakes so I deciced to stop using it.
Get a m5 pro mac with 48gb vram and you’re good to go for a few years and that computer can be used for others purpose as well
Dog slow, at last for me. I can't crack 30 tokens/s. I'm trying an Intel B70 next.
How do you do the cctv analysis ?
It’s combined with UniFi I assume, what else has good API, Hooks, or consumer level integration to play around with with.
Frigate is an excellent open alternative.
> Agents are the new UI
Funny as I'm using LLMs now to write UIs.
> (...) it hums at120t/s. It’s really a surprisingly good model for general tasks, and I’m soon to see how it does with more agentic kit.
This is exactly why I interpret the aggressive price cutting by OpenAI on their Luna model line as a clear sign they are scared shitless of local models, and how they can trigger a "the king has no clothes" moment for AI companies like OpenAI.
> as a clear sign they are scared shitless of local models,
The number of people with hardware to run a model like that is very small.
It's going to be many years before hardware to run medium size local models is widely available. Running mid-size local models doesn't even register as a factor in pricing decisions right now.
It's such a small part of the market and will be for many years, until RAM and other hardware prices come down.
> The number of people with hardware to run a model like that is very small.
True, but the number of people who pay the non-subsidized API price is also very small, and it's all in big companies who would have no issue buying or renting hardware for that purpose.
Ram prices suggest there is going to be a mighty glut of the stuff in a couple of years. We may easily see 32->64->128 GB system RAM off the back of that and CPUs seem to be moving to unifying VRAM and main memory.
I don't think running a ~200B parameter model is that out of reach, it might be usual by the mid 2030s for people in the tech world. Factor in the crazy improvements in quality given a number of parameters too.
1 reply →
but the number of small to medium sized companies which really want to run (some of) the AI usage under their control and are willing to drop some 100ks on it aren't that few (especial for non US companies given the state the US is in)
1 reply →
> The number of people with hardware to run a model like that is very small.
But there are a lot of companies who do and rent it out as a service.
Idk, I’m running 180b on $2500 of hardware at 120 T/S (single user of course) but I bought it before the clown show. Now it’s probably more like $10k lol
I expect I’ll be able to run 1/2 terabyte models in 5 years for a couple $K in hardware. Inference is going to be a lot like Bitcoin mining. Only the peak of the power/performance curve is going to be profitable. Which means a glut of surplus hardware that can’t make money but makes tons of tokens.
1 reply →
Many years is only long in relation to the AI hype cycle. In terms of a human lifespan it is not that much, maybe 10 years at most and that's pessimistic. We technically live in that future already it is just very unevenly distributed.
1 reply →
> The number of people with hardware to run a model like that is very small.
That's not the problem anyone is facing, and analysis anyone is doing. At all.
The problem is the huge AI-related budget that companies who dove head first onto AI are facing.
Therefore the problem everyone is faced with boils down to whether they keep paying the likes of OpenAI a de-facto business tax, or they replace that tax with a one-off CapEx invoice followed by a utilities bill.
And you don't need a frontier model to do most things. In fact, the biggest challenge in using local models in agentic coding is the models supporting these agent-based tools.
> It's going to be many years before hardware to run medium size local models is widely available.
Nonsense. Speaking for myself, my employer already introduced a hard cap on AI credits while rolling out hardware support for local models, and even went to the extents of rolling out workstations with 64GB of RAM as a baseline. Consequently, we are using the likes of gemma4 for a fair share of everyday tasks. This has a transformative effect on your workflows because suddenly you don't ration tokens, and agent orchestration doesn't become a cash guzzler. So far the only constraints are tool support and token throughput, but that's not a concern.
1 reply →
[dead]
7 replies →
At cost of power at $.45kwh here and dual 3090 setup easily approaching 1kw per hour at 120 tok/sec, latest Luna is cheaper per million tokens.
Your electrical costs are likely less than mine though
A dual 3090 system shouldn't go above 600W wall power while running inference, even less with some weights in RAM as you need to do with "only" 48GB VRAM and this model. Unless you are running water cooling, the cards are likely sitting at 200-250W power limit each anyway so they don't run too hot. This is considering a consumer tier CPU and mainboard with about 90W TDP, not a dual CPU or something like a threadripper that can easily push the power draw to 1kw if it runs the calculations on the RAM weights. My dual 3090 setup usually drew about 200W-300W average with LLM workloads according to my HA historical data.
I'm working off a 128gb Strix system on mostly hydro and it's worth it for me. Qwen 3.8 27B and Flash Next, Gemma 4 and MOE models. It Keeps my thinking and methods local while moving at a speed I can actually read and think about.
It can be annoying when you're running something long and realize there is an issue. But hey, I'm not wasting paid tokens. Reset and try again. I've probably thought of a better way by then too.
Yeah electric costs are nil on the microgrid but capacity is limited.
> as a clear sign they are scared shitless of local models
Partially agree and partially disagree. Qwen 3.8 next (120b + 50b ngrams) is at the upper end of a local model, and serving it at good speeds (esp for pp) is not cheap. So no, oAI are not particularly scared that everyone will suddenly start using local models of this size / quality.
I do agree that having these models in the open is great pressure on the price point of "small" API models (like luna, haiku, etc). And that's good news for us consumers and for the field in general - more competition is always better.
And on the cost estimate, the true power of local models is the guarantee of running them on prem w/o any data going out. Price wise it is still cheaper to go API, for small teams.
The "upper end" of local models as of today would be something like MiMo 2.6 Pro or Kimi K3, either of which is significantly smarter than Qwen Flash-Next. Roughly comparable in smarts to the old GPT Terra series, not quite as capable as Sonnet. But yes, running these models locally at realtime-usable speeds and without heavy quantization would be even harder. They can be quite tolerable for plain Q&A chat and great for running multiple unattended workloads on a 24/7 basis, but that may be a bit of a niche.
2 replies →
Viewed differently, they make it much cheaper to distill their models.
With the 2x3090s, do you have them connected with the NVLink connector or are they just running over the PCIe lanes?
iirc 3090's doesn't have NVLink, or at least the one I have doesn't/
It must be the one you have, because 3090's were basically the last consumer level card to have nvlink.
https://www.fibermall.com/blog/nvlink-compatible-cards.htm
> These are going to be so awesome to buy in 5-8 years!
Literally my first thought!
Are you running club-3090 or something else?
Flash-Next runs on Strata, version 0.1.42, an inference engine built specifically for Qwen3.8-flash-next