Comment by jacquesm
6 hours ago
It's absolutely true. I have three different setups here, a rig with 8 gpus that I use in either one of two modes (all 8, very heavy model or just one much lighter) and two DGX sparks. I haven't used a paid model in months. Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct). But the fact that I'm not sharing my work with the paid providers is much more to my liking than having big brother looking over my shoulder at every step and occasionally giving me the 'I can't let you do that, Jacques' treatment.
Gaming used to be done in arcades for 25c a play*.
Over time NeoGeo realized they could sell you the hardware to play at home (“on-prem”) at arcade quality albeit at a price out of reach to most people. Alongside them, Nintendo and Atari gave you 80% of the fun at 20% of the cost. History will view games arcades as a flash in the pan compared to gaming at home.
Namco, Capcom, and Konami went into selling software instead of hardware but it feels hard to imagine buying a home copy of Fable** for $X,000 any time soon.
⸻
* That’s for video games. In fairground arcades you’d literally pay for tokens then use tokens for play. The analogy almost writes itself.
** The model, not the video game.
If you think the idea of buying Fable like model is hard, you should really try some of the latest open models like Qwen 3.8. They’re getting there very fast.
Qwen 3.8 and GLM 5.3 are my workhorses right now. Qwen for simpler stuff because it is wicked fast using 'ninfer', GLM 5.3 because it is very good a more complex stuff and less likely to get stuck so I can let it run for much longer without intervention.
1 reply →
How much did you shell out for this setup? What’s the TTFT and the output tokens/sec rate? How much electricity does it consume?
Working on a blog post to document it all, but that will take a few more days. The short version: the big rig is probably $25K, ttft is dependent on how much prefill you have but typically a couple of seconds (2K prefill/sec in one setup, about 500 in the other), tgs 100 to 150 or so depending on what it is doing and how well the speculative decoder works with the current context. Electricity is pretty bad, full tilt it's about 2500W or so but I've power limited it to 1500 to keep it civil. It is not cheaper than using a frontier lab (or at least, I don't think it is), but the independence bit is the important part for me and I don't care if that carries a surcharge, even if it is a substantial one.
I'm not saying your approach is wrong - but for my workloads, it makes a lot more sense to just pay for a decade (!!!) of Claude Code 20x Max.
41 replies →
The benefit of the agent working firstly for you rather than for Anthropic or OpenAI will become increasingly apparent. We're already seeing signs: remember when Anthropic openly promised to have Claude sabotage people using it to create competitors? And OpenAI is looking for revenue opportunities from advertising, which can't end well for their users.
Just think of how user-hostile other online services have gotten, and imagine how acutely painful the proprietary LLM providers could make theirs. Thankfully, open models provide some counterbalance.
Did you get insurance on all this stuff? I imagine home insurance would not qualify for a bespoke home-datacenter.
> It is not cheaper than using a frontier lab
Are you comparing to subscription prices or API prices? Because if it's the former it won't last forever.
1 reply →
I mean, I just have 4xV100S GPU and a Macbook Pro w/48GB of RAM and I find it kind of amazing.
My server can run Qwen Flash Next at 500 t/s aggregate (about 200 t/s single user). It's pretty great. I use it for local coding and all my worker agents. It is about as reliable as Sonnet 5 for agentic coding. I make 3D models, keep the home network and server updated, fully built ansible files.
You do NOT need 20K in hardware to run these models reliably. I built my own inference engine from scratch for fun. You can run Strata on 16-32GB cards at very respectable token rates if you have the standard RAM to go with it. I strongly encourage people to go use this model on OpenRouter and try it out for tasks.
I also have a somewhat lobotimized quant of GLM Flash 5.3 working and it's pretty decent too.
V100S is a very good card for some workloads. And they were cheap as well.
I have 240G VRAM total which is a good reason why that rig cost as much as it does, and if I were to build it at today's prices it would be even more expensive.
2 replies →
[dead]
At work two different departments have it on their radar to do - apparently there are GPUs waiting to be used for something like it.
Smart move to get them bought ahead of time.
8 gpus is cheap now?
Well... that's a tricky one. I bought a lot of these second hand. I can sell them for more than they originally cost new, so indeed, it is not 'cheap'.
Then again, I run a business and for the business it is absolutely manageable, it is roughly the price of a single job. Privately this would be a different matter but I assume that if you would do this privately you'd simply get a single (but powerful) GPU second hand and offload to RAM. Not as fast but with llama.cpp, 'ninfer', DwarfStar and various other clones of those two you can get some pretty decent performance out of single GPUs.
I've been buying powerful hardware since the 90's, starting with SGI and then ever larger Linux boxes. I make my living with them so it's never been a huge decision to invest some of the proceeds back into the tools. But on an absolute scale, yes, it is a lot of money.
Depends on context of what people build.
I mean IBM already has a track record of building gigantic amounts of garbage and over charging large corps and govts for it coasting on their brand name and networks to get deals. So using some cheap ass model fits their MO. Having used their shit mountain called websphere, what an IBM CEO says is not something I take too seriously.
IBM has two faces. Their consultancy arm is the exact opposite of what I would call a good company, even so, they make a good bit of money. The hardware side of the business is still quite impressive, yes, it is too expensive but their power rack boxes are effectively a little mainframe (check the IO stats on those things) and the performance is - I have to admit - impressive. That said I would never buy it because I can get much better value elsewhere. Even so I think they could do this if they really wanted to.
They do have big advantanges because many large corps have become totally dependent on them having outsourced everything and lost internal knowledge. That internal ops knowledge is very crucial/useful to take full advatange of what AI offers. In the coming years much of it will leak into frontier models. After which corps might not need IBM/TCS etc and just rebuild internal teams to handle ops like they once did.
> Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct)
I can't wait to use models that are 'slightly less good' when I am at work instead of something like Fable that lets me focus on the problem at hand as much as possible rather than also spending additional effort correcting for the tool I am using...
You need to reread what was written. No one is proposing to take away your right to leak proprietary edge to the rapacious frontier labs that need to eat most of the economy in order to survive.