Comment by justrunitlocal
18 hours ago
We've been running our 10 dev org on 8 H100s on open models (with some tweaks). Sure they aren't as good as the big providers but they 1. don't go down 2. have pretty damn high tok/s. It pays for itself.
Posting with a fresh account because I'm not supposed to share these details for obvious reason. If you want help on setting this up, just reply with a way to reach you.
First of all 1) 8 H100S are NOT ENOUGH for today's premier models (500B+ tokens) and if you do run a obsolete model forget about memory.
2) After buying the 300k GPUs, your electricity cost will put you in competition with hosting on cloud costs, you will probably lose dollars this way.
3) NVIDIA will charge you a kidney to provide driver/hardware support if anything goes wrong.
This inherently a bad idea and this person is probably trying to promote his startup.
yea just buy 300k worth of hardware and bob's your uncle
It was pretty hard to justify the purchase to the board but we got a decent deal from a nearby data-center (~15% discount). Thankfully, it's fixed cost, its an asset we can use for our taxes, and it will survive for years to come. The only thing we have to work on is maintenance as well as looking into some renewable energy options.
We're also looking into how to do some secure cost sharing with this so that all people need to pay for are what it costs for us to run everything! We're just planning on reserving at least 51% of the capacity for us and the rest for everyone else.
Sorry, didn't mean to be dismissive, I was just being a dickhead needlessly.
I actually respect this a ton, good work.
3 replies →
One dev's salary to give a 10 person team unlimited approximately free agentic coding for the foreseeable future, plus privacy.
And another salary to have someone set up and run it
We're planning to do the same thing - buy something like 8xH100 and run all coding there. The CTO almost agreed to find the budget for it but I need to make sure there are no risks before we buy (i.e. it's a viable/usable setup for professional AI-assisted coding)
Can you share what models you run and find best performing for this setup? That would help a lot. I already run a smaller AI server in the office but only 32b models fit there. I already have experience optimizing inference, I'm just interested what models you think are great for 8xH100 for coding, I'll figure out the details how to fit it :)
8 x h100 80's don't give you enough to run the latest 1tn + parameter models (especially at the context window lengths to be competitive with the frontier models)
Verda has B300 clusters, 8 for USD $55/hour in 10 minute billing blocks
Check out Verda you can rent whatever super powerful GPU clusters you need in 10 minute increments. Deploy any open weight model using SGLang and away you go
Deepseek, GLM, Minimax or Kimi are the most likely contenders.
I’ve been using kimi 2.5/2.6 for the past 2 weeks and it’s really not far off OpenAI and Claude models. I am a coder so it’s not all vibes but I am definitely more in the “spec to code” mode than “edit this file for me” and it copes just fine. Needs a bit more supervision than the frontier models but it’s also significantly cheaper. If I were anthropic I’d be shitting myself, their prices are going to 10x over the next 2 years
1 reply →
> Sure they aren't as good as the big providers
If you haven't done so already, finetune the model on all your company's code that you can get your hands on. This is one of the great advantages that you get when running local models. I like the style of the generated code much better now, I have to rewrite much less, and my prompts can be shorter too. But maybe these already are the "tweaks" that you mentioned.
How would they do that? Would it be as easy as telling a model "Hey, review all this code, identify patterns, and then write in this style going forward"?
Sorry if this is a stupid question, I've never finetuned or trained a LLM.
Unsloth has consumer accessible stuff on fine tuning models
This is the actual answer. Man I hope to find a company like yours sometime soon. I am sick of all the issues with having 3rd party IP generation