Comment by echelon
9 hours ago
Wouldn't that be crazy - a hobbyist market for H100s?
Maybe someone could start a business buying up and rehousing these.
9 hours ago
Wouldn't that be crazy - a hobbyist market for H100s?
Maybe someone could start a business buying up and rehousing these.
They're pretty specific to the datacenter use case with no outputs and they need to be cooled externally, principally through the very loud high speed fans used in data centers. I suppose you could strap a fan to one and put it in a normal case or maybe make a dedicated after market cooler (like the water blocks made for water cooling cases).
I think there's a market for a home AI server that can run an LLM or video gen model behind a web frontend. Not literally a raspberry pi strapped to an H100, but something with lopsided enough specs that people joke it is.
And precisely because it's such a huge headache to do yourself, I think a small company could make a nice business wrapping up used datacenter cards in that sort of server.
I'm doubtful they can, based on current inference prices. An H100 draws about 50W while idling, which is ~$8/month at average US electricity prices. They also draw ~400W while active.
The electricity prices are relevant because if you paid $0 for your H100 and didn't use it a single time, you could buy millions of tokens in inference just on the electricity it draws while idling. If you can't keep that thing saturated through the night, you're probably underwater overnight. Likewise, it's too small to run even the frontier open source models so you need to be able to live with worse models.
Max power matters because you aren't going to run many of those H100s before you blow breakers in most houses. Newer houses in the US are 15A service to non-kitchen breakers, so 1650W (that might be peak rather than continuous, not sure). If you're plugging that into an existing run, you could maybe run 2 before you start blowing breakers? You can't just plug 4 H100s into the wall in a normal house.
Maybe I'm wrong, though. I'd be curious, it'd be neat to run my own inference for something more than what'll run on a 3080.
1 reply →
Meh
Models are too big now
This is also true of the older P100 and hobbyists do this stuff now, today
If you check eBay for V100 SXM variant, you will often see them sold in combination with a cold plate, together with a PCIe carrier board. There’s definitely a market for them.
> Very loud high speed fans
if you haven't heard a 5u server intended for a datacenter rack come to life it's quite the experience. Sounds like a plane taking off.
My workstations are usually tower servers, which are the same design as their 4 and 5u rack counterparts.
Until the thermal management kicks in, they sound like jet planes. When the thermal management starts and assesses the required cooling, it’ll throttle down the fans to reasonable levels.
That is, until the moment you push the machine to its limits. When then happens, you might get back to the same levels of the boot time, but it’ll require you to push everything to the max - CPU, memory, storage (all 24 bays) and so on. For a normal user, there is a lot of room and it’s virtually impossible, even with a dozen of Teams windows open.
1 reply →
"Sounds like a plane taking off."
Helicopter. I live in Yeovil, Somerset, UK - there's a helicopter factory just down the road. I had a IBM "AS/400" or whatever they are called now in our computer room rack for a customer and it made nearly as much noise as everything else put together. It was clearly tuned for start up noise to impress because they would fire up in sequence, rise to a crescendo and then slow down in sequence to just a din instead of painfully loud.
A switch or PC server on boot will normally run up cooling fans instantly to max as a default protection mechanism until the "OS" has started and sensors read and then the fans will slow down to deal with the actual thermal load.
If you switch off your air conn, it gets noisy, quickly. Recently in the UK we are seeing routine temperatures around 30C and we broke 200 odd year records for temperatures a few weeks back. I know its even worse elsewhere but our infrastructure is not designed for this. Here we are at the same latitude as Calgary AB!
Closest thing I can think of is the old LucasFilm "THX" trailer. Only there's no release segment, not until the BMC comes up a minute or so later.
Or you can secure yourself a DLC one and try to feed it with the correct regime (liquid composition, temperature and flow rate). I'd say good luck.
These things get hot and are fussy about their requirements.
H100s (and the PCIe converter card you need) are available on eBay.
The GPU alone has a TDP of 700W, together with everything else (CPU, RAM, storage, fans) you're looking at 1500W+. Depending on the country, that may be enough to saturate your home's electricity uplink...
That's actually less bad than I assumed. High-end gaming GPUs are already almost at 600W with peak usage above that. I assumed it was much worse than that; that seems absolutely feasible for running at home.
Never heard electrical uplink before but to those wondering Italy, India, and Japan all have requirements around 3kw. I was slightly surprised by this, but in the end if your running this in a tiny space with that low of power your already probably not buying used H100s.
Hell no. 1.5kw is half a socket capacity. A heat pump or many kitchen appliances are using much more. Our car charger uses more as well.
That's why they said "depending on country"
In North America we are on 120V, making a standard 15A outlet only 1500W max, and something like 1200W sustained. To use higher wattage appliances, we have to upgrade our outlets to 20A (2000/1600W) or up our voltage to 240V, but that carries a different set of plugs and outlets as well.
2 replies →
In the US they have a 120V system so it might actually saturate a normal socket over there. My PC room has a 16A fuse and 230V though, so should be plenty :)
A typical hair dryer uses about 1500W. I guess GPUs are mechanically not much different from a hair dryer.
Install it in your bathroom, use it like a restroom hand dryer.
1 reply →