Comment by tyfon
9 hours ago
I half expect Nvidia to have buyback contacts like Ferrari with the larger customers to prevent a price crash when they all upgrade and to keep them scarce.
I hope not though, perhaps I can pick up a H100 in a few years if they get sold on the open market.
I worked at an org that had a substantial on-prem GPU datacenter. We transitioned to <Big Cloud Provider> with a substantial negotiated discount rate, with part of the contract being we would sell them all of our hardware and not purchase any more.
> and not purchase any more.
why would anyone sign such a contract?
If you're planning to run in clouds, committing to not buy hardware (during the contract term, presumably) isn't a big imposition. Maybe you switch to a different cloud, and you wouldn't buy hardware for that.
If you want to switch back to on prem, there's probably a way to structure acquiring hardware so it doesn't break the contract. Maybe you lease it, maybe the purchase happens through a related company, maybe there was no way for the contracted cloud to find out...
The usual very short term corporate thinking that maximizes quarter profits while bankrupting the company in the long term. IMO a very shortsighted decision, if not downright stupid.
I don't know if this was IT shrugging me off or if it was something real but some IT person at this big ISP I worked at told me that they cannot just buy an SSD — my windows box at work was running off of a hard disk in 2019 — and that there was some contract that said any computer hardware we bought had to be through HP or something like that and it takes many months it something like that.
1 reply →
How does this work? Is the OEM giving the cloud provider a big discount or is the cloud provider giving you a teaser rate to lock up your business.
The list price of cloud is 10x higher than the price of hardware itself so that leaves room for some discounting.
2 replies →
There's going to be a golden age of GPGPU compute in the next few years once A100/H100 are fully obsolete for running frontier models efficiently and the price plummets
It will be perfect for stuff like GPU-accelerated query engines, "classical ML" and every other CPU-based workload that could conceivably be offloaded to GPU
There's nothing stopping you doing this now.
You can get used 16GB P100s on AliExpress for ~$100 if you want obsolete GPUs. Allegedly new AMD BC 250s are only slightly more.
I've looked at this some but I already have a GTX1070 which is only supported upto CUDA 11.9.
That's precludes some interesting modern optimizations out of the box. I've spend a lot of LLM tokens backporting some things, but I'm really not sure the hassle is worth it.
New hardware is just better. I think in maybe 5 years when supply and demand are back in equilibrium we are going to have some killer technology for decent prices, and 15yo H100s won't look attractive.
GPGPU? General Purpose GPU? If embarrassingly parallel CPU algorithms weren't offloaded to the GPU previously, why would the A100/H100 price drop make a difference? We had cheap GPU in the past and we still left plenty of performance on the table with CPU programs because they were easier to build.
Is the idea that previously maintaining GPU programs was expensive whereas now AI makes it cheap? If so, I could buy that line of reasoning.
Maybe relatedly, I expect (hope) the hardware manufacturers will ramp up supply in the meanwhile which would also put downward pressure on GPUs. Right now though this hardware crunch is making me sad, not even because of GPUs but also because of general memory / disk.
GPGPU programming has become significantly easier now and the payoff is bigger (better hardware), due to the immense investment in this due to ML/AI.
1 reply →
We've not had cheap GPUs with this much VRAM before, though. Might be an interesting change, though I also doubt it personally.
1 reply →
The only reason a datacenter would ditch their H100 is if it becomes uneconomical to run them, with newer silicon providing much more power efficiency. When that happens they'll look like a used V100 looks today: horribly inefficient, lacking modern data types and engines, requiring screaming server fans with weird adapters to not melt, way beyond end of life in terms of cuda support. Almost completely damn useless unless you really have no other alternative.
[dead]
That should be illegal. Sounds like a very fraudulent business tactic.
Fraudulent, I wouldn't say so. Anti-consumer or anti-competitive? Sounds like.
The hyperscalers signing these contracts have decent legal departments. Think about Oracle for example - I'm pretty sure they know every trick there is about beneficial contract drafting.
I don't think they need some special protection against this kind of contract.
It’s common for car companies when they enter a new market. It removes uncertainty from the second hand market.
By doing that, you know upfront what the value of your used hardware will be at the time you decommission it. It removes a lot of the risk for buyers in a volatile market.
Would you call a trade-in a fraudulent tactic?
Trade ins are fine. A contractual commitment to not buy hardware sounds like illegal restraint of trade.
Trade-ins are voluntary.
6 replies →
The Grift Economy places all legalities on the marks and their inability to form legal fights.
Sounds like capitalism at its finest.
Good old "win-win-lose"
Wouldn't that be crazy - a hobbyist market for H100s?
Maybe someone could start a business buying up and rehousing these.
They're pretty specific to the datacenter use case with no outputs and they need to be cooled externally, principally through the very loud high speed fans used in data centers. I suppose you could strap a fan to one and put it in a normal case or maybe make a dedicated after market cooler (like the water blocks made for water cooling cases).
I think there's a market for a home AI server that can run an LLM or video gen model behind a web frontend. Not literally a raspberry pi strapped to an H100, but something with lopsided enough specs that people joke it is.
And precisely because it's such a huge headache to do yourself, I think a small company could make a nice business wrapping up used datacenter cards in that sort of server.
3 replies →
This is also true of the older P100 and hobbyists do this stuff now, today
If you check eBay for V100 SXM variant, you will often see them sold in combination with a cold plate, together with a PCIe carrier board. There’s definitely a market for them.
> Very loud high speed fans
if you haven't heard a 5u server intended for a datacenter rack come to life it's quite the experience. Sounds like a plane taking off.
4 replies →
Or you can secure yourself a DLC one and try to feed it with the correct regime (liquid composition, temperature and flow rate). I'd say good luck.
These things get hot and are fussy about their requirements.
H100s (and the PCIe converter card you need) are available on eBay.
The GPU alone has a TDP of 700W, together with everything else (CPU, RAM, storage, fans) you're looking at 1500W+. Depending on the country, that may be enough to saturate your home's electricity uplink...
That's actually less bad than I assumed. High-end gaming GPUs are already almost at 600W with peak usage above that. I assumed it was much worse than that; that seems absolutely feasible for running at home.
Never heard electrical uplink before but to those wondering Italy, India, and Japan all have requirements around 3kw. I was slightly surprised by this, but in the end if your running this in a tiny space with that low of power your already probably not buying used H100s.
Hell no. 1.5kw is half a socket capacity. A heat pump or many kitchen appliances are using much more. Our car charger uses more as well.
4 replies →
A typical hair dryer uses about 1500W. I guess GPUs are mechanically not much different from a hair dryer.
2 replies →
isn't Ferrari the brand that requires any purchaser to be an existing owner? I could see NVIDIA going for something like that.
Invite-only is only for special edition hypercars and halo models. Standard models can be purchased by anyone with the funds and desire.
Porsche is doing that for RS cars.
Yeah and now they have lost 98% of their profits in the last year. How's that catering to the ultra-wealthy working out for them? They used to build attainable cars that were nearly as cheap as a Corvette. Now they're double the price, and most certainly aren't double the performance.
1 reply →