Comment by octoclaw
2 months ago
Running a small project on Hetzner from Germany. Got the email this morning. Honestly, even after the increase their dedicated boxes are still absurdly cheap compared to what you'd pay at AWS or GCP for equivalent specs.
The real story here isn't Hetzner being greedy. It's that AI companies are vacuuming up every DRAM chip on the planet and the rest of us get to pay the tax. I priced out a RAM upgrade for my home server last week. Same kit I bought 8 months ago for 90 EUR is now 400+. That's not normal market dynamics.
What worries me more is the second-order effects. Startups that would normally spin up cheap VPS instances to prototype and iterate now face meaningfully higher costs at the exact stage where every euro matters. The "just deploy it" culture that made European indie dev scene so productive was built on sub-10 EUR/month boxes. Those days might be over for a while.
"The real story here isn't Hetzner being greedy. It's that AI companies are vacuuming up every DRAM chip on the planet and the rest of us get to pay the tax."
We might also have our aquifers depleted and our electricity prices skyrocket. But at least we see really great benefits, such as being able to script some side-project while unemployed due to AI.
>aquifers depleted
Oh it's this thoroughly debunked talking point again.
https://andymasley.substack.com/p/the-ai-water-issue-is-fake
I just can't believe how HN turned into disinformation / propaganda machine over last few years. Pretty much every topic is politics and disconnected from reality.
1 reply →
Anyone who thinks modern data centers don’t use recirculated water can safely have their opinions summarily discarded.
Data centers consume...a lot...of water by design, recirculated water, does not means no water consumption. Water must be continuously added in evaporative cooling systems used by many data centers.
[1] - Cooling towers reject heat through evaporation, which uses water, not just recirculates it. Evaporated water is lost to the atmosphere and must be replaced with "make-up" water. As a result, recirculating cooling loops still require new water input to make up evaporation and blowdown losses.
[1] - https://en.wikipedia.org/wiki/Cooling_tower
1 reply →
Anyone who thinks that modern data centers don't evaporate their "recirculated FRESH water" straight into the ocean can safely have their opinions summarily discarded.
Please google "datacenter evaporative cooling" and then re-evaluate
3 replies →
Grim
> It's that AI companies are vacuuming up every DRAM chip on the planet and the rest of us get to pay the tax.
DRAM is priced based on supply and demand, like every other market.
When demand goes up, the price goes up for everyone. It’s not a “tax” on the rest of us in any sense. There’s just a lot of demand everywhere.
> That's not normal market dynamics.
This is actually a textbook example of markets functioning in response to a demand shock where supply cannot be increased rapidly.
I do find it interesting that so many people think “market rate” means the opposite of what economics teaches, and that prices should stay stable and not change much when the economic conditions change.
I also find it interesting to read all of the “we shouldn’t let them…” takes in response to this situation. The DRAM market is international. Trying to restrict it in one country would just see the data centers get built in another country.
Saying this is just the market...is like saying housing is a free market after hedge funds buy your entire neighborhood...
But... They're not wrong. That IS the market. Unrestricted, gloriously free market with its historically predictable outcomes - yay!
That's not where the interesting discussion is. The interesting discussion is with the notion that free unregulated markets are universally good and will naturally lead to positive outcomes because... I don't know, I'm personally not religious, but somebody here will help me :-).
22 replies →
People love to say that but they own a very small percentage of housing in reality. What’s driving housing costs is also supply and demand. Especially supply, since we’re not allowed to build any houses in most places people want to live.
7 replies →
Hedge funds don’t have as high of institutional ownership as you assume. It’s actually pretty small.
That said, nothing about the situation you described is at odds with “free market”. You’re describing the operation of a free market.
I think a lot of people want “free market” to mean the opposite: A highly restricted market where they are protected from any supply and demand inputs from anyone else. They just want cheap things and don’t want to compete with anyone.
There are two sides to a free market, though. In your example where a hedge fund comes in and buys your entire neighborhood, they would have to do so by outbidding everyone. This drives up the price. If it’s an economically irrational move you’d be smart to sell your house to them at an inflated rate, too! Then move back in when the prices crash down.
Which neighborhoods are now entirely owned by hedge funds?
5 replies →
Are there regulatory barriers to entering the DRAM market like there are in housing?
1 reply →
correct, both of those things are examples of free markets
6 replies →
> When demand goes up, the price goes up for everyone. It’s not a “tax” on the rest of us in any sense. There’s just a lot of demand everywhere.
> This is actually a textbook example of markets functioning in response to a demand shock where supply cannot be increased rapidly.
You act like it's a competitive market. It's not the case. It's an oligopoly with an extremely inelastic supply side.
The market is already completely broken and ineffective due to concentration and export controls. The actual response to a major demand shock should be investments to increase capacities but it's currently extremely limited because suppliers want to protect their margins and fear the market contracting again.
> It’s not a “tax” on the rest of us in any sense. There’s just a lot of demand everywhere.
Curious on whether you will still hold your stance if OpenAI gets a taxpayer bailout. Even disregarding a bailout, they are already lobbying hard for tax credit expansion.
A government bailout of OpenAI would be a regressive redistribution of wealth to some of the least needy people in all of society, which is a horrendously poor use of government funds. But that has no bearing on the fact that calling high DRAM prices induced by high demand a “tax” stretches the meaning of the word beyond all recognition.
There are many horrible things in the world and we don’t need to label them all as a “tax.” If we use words in an imprecise way, it obfuscates the truth.
1 reply →
Most markets don't have three purchasers trying to corner the entire supply of one product.
I guess we’d call this oligopsony?
The monopsony (single buyer of a good) equivalent of an oligopoly?
Phew, it is a word, but not a highly studied one!
https://en.wikipedia.org/wiki/Oligopsony
Economic history is full of examples of demand shocks. This is not some unique situation that has never occurred before.
This is actually a clean commodity price spike because it’s specifically not for market manipulation or financial engineering. It’s because demand for this product really did explode overnight.
6 replies →
> This is actually a textbook example of markets functioning in response to a demand shock where supply cannot be increased rapidly.
The problem is that demand is being propped up by speculative capital. The AI companies are a bubble that is suffocating productive parts of the market with the hording of capital which they're now using to also hoard hardware. All this without making money for data centres that aren't build yet, for a handwavy promise that an AGI will magically solve all the worlds problems.
This is not normal, and it is not good for the broader economy.
Yeah the dudes argument is bunk when we remember that openAI bought CAPACITY and not actual product. The market is also heavily manipulated by the big 3 players in the market.
OpenAI brazenly used their market position to create artificial scarcity. That's not normal market behavior. That's manipulation. And now we all suffer.
[dead]
> Trying to restrict it in one country would just see the data centers get built in another country.
I'm surprised this isn't already what's being done. Inference doesn't require super low latency with the client, and the population's support of AI (and especially data centers for it) is waning quickly. This feels like another ideal use case for outsourcing the stuff Americans don't want to see to somewhere that it'll be someone else's problem.
> Trying to restrict it in one country would just see the data centers get built in another country.
Sounds like not stressing the electricity infrastructure in Spain, to run inference for Facebook North American posts, should be seen as a positive...
Factory capacity does not follow market dynamics easily
I think the usefulness of market dynamics is their ability to follow things like factory capacity, which are themselves hard to follow, not the other way around.
1 reply →
Can't agree more. We can also predict with some confidence that in a year or two, supply would have adjusted and ram will be cheaper in the long run. We benefit from the expanded demand even if the fact that it first lands as a shock is disruptive to prices.
GPU prices went through the roof for crypto and then the pandemic and never really recovered to pre-pandemic prices before once again spiking because of AI demand. So where's the increased supply of Nvidia cards to account for all the continued demand? And why haven't RAM manufacturers announced plans for increased production (instead of pulling out of the consumer market altogether)?
The past 6 years of GPU pricing (the 5080 launched at $1000, currently $1500-1800 at Microcenter) don't exactly fill me with confidence that RAM manufacturers will increase supply to meet demand and bring down prices again.
If the last five or so years are to go by, we'll just have another pricing shock by then. So yay.
The problem is that OpenAI has cornered the market. Maybe they haven't crossed the legal line or more to the point no one in this corrupt and incompetent administration is going to prosecute them, but buying up 40% of a market which hasn't got any additional capacity is cornering by any measure.
So yes, this is not a normal market. Your claim of a functioning market is the same as saying my laptop, having lit on fire, is a functioning computer after having 10,000 volts applied across it.
> DRAM is priced based on supply and demand, like every other market.
Please don't explain it away like that - you are referring to the theoretical "ideal" market where a bunch of small companies compete with low margins to the benefit of the wider customer base. This is not what is happening. We have a couple of intrinsically worthless, LLM-whale companies, working literally to swallow and entshittify literally everything in their weird transhumanist/accelerationist/weirdo way. To add to the insult, the whole creation of artificial scarcity is almost a political construct, paid for with "monopoly-the-game-money" that these companies DO NOT EARN but instead BORROW based on vague and dishonest promises of achieving a "Country of PhDs in a datacenter"/"Pocket PhDs"/"AGI by 2025" (oops, now apparently by 2028 according to the OpenAI CEO). In their weird vision, as humans we should be merely cattle to be managed, not independent spirits with interest and aspirations. That ghoul Karpathy speaks about "ghost in the machine", overlooking the magnificence of the already existing "ghost in the machine" in the form of human beings. We should not have to swallow the increasingly crappier future these folks are insisting on pushing on all of us.
In many countries, its illegal to manipulate prices in bulk.
What makes it manipulation? If 5 companies want to buy a quadrilion ram chips to build datacenters, why is this manipulation moreso than a million companies each wanting to buy 100 ram chips?
I think the problem is that both the buyers and producers are too large. Governments should not allow companies to become this big, because... <gestures broadly at everything>. If there were a thousand ram makers and a thousand datacenter builders, this particular problem would not exist.
But you can't just label any price evolution you dislike as "price manipulation".
4 replies →
AI demand is subsidized by the bubble. Those operators buying the RAM are not paying using money that exists. Market economics are not working here.
"it happened therefore it's normal"
But aren't those the same startups that think they need to run on AWS EKS instead of using a single cheap server? The cheapest used Hetzner server currently is €39.24 / month:
- Intel Core i7-6700 - 32 GB - 2 x 480 GB Datacenter SSD - 1 GB/s - 20 TB traffic
Their VPS are even cheaper. And you can run a lot on this.
Similar to my favourite OVH servers, but I have unlimited traffic at 0.5Gb/s 64gb ram and dual mics. Similar price (with vat in Poland).
If you wanted to run same workloads on Aws it would cost you few hundred euro a month.
I see a silver lining to all this. At least maybe the silly "throw more horizontal scaling at it" will stop being a default response to all performance problems and people that are able to squeeze more processing out of the same hardware will be sought after again.
If your only need is a lot of bandwidth with very low server CPU use that’s fine.
That CPU is ancient, though. Over a decade old. That DRAM is 2-channel DDR3.
This could be a good deal for someone, but entrusting your startup’s operations to a 10 year old slow computer in Germany instead of using EKS would be an extremely short sighted move. A startup should be developing software and shipping it quickly to validate the market, not pinching pennies to save the equivalent of a couple hours of developer salary.
I said the same thing to myself.
But then I remembered that what AWS gives you is the same generation of CPU, just obfuscated.
GCP Also obfuscates it, but not as much: https://docs.cloud.google.com/compute/docs/general-purpose-m...
(note: skylake is 10 years old)
>That CPU is ancient, though. Over a decade old.
Coincidentally so are the t3 / t3a instances on AWS that everyone loves to use especially for dev/staging environments
2 replies →
I would guess that 99.9% of startups wouldn't notice the age of the CPU if they aren't in the business for CPU compute power.
Also, if you don't want to provision software systems, you probably shouldn't use Kubernetes at all. Both this and compute are niche businesses and neither would rent a budget server anyway.
> That CPU is ancient, though. Over a decade old. That DRAM is 2-channel DDR3.
6700 should be DDR4 unless they're using some weird-ass setup.
While I agree with the last sentence, I would suggest you buy what is needed not what is latest.
Except 40€ a month is extremely poor value for this CPU that's more than a decade old.
Is it? It's 10x the price on GCP:
https://cloud.google.com/products/calculator?dl=CjhDaVEyWWpJ...
The CPU is the same generation; https://docs.cloud.google.com/compute/docs/general-purpose-m...
No, that's actually a really good deal for dedicated hardware with those specs. For a project sized for hardware like that, the CPU is a lot less relevant than the RAM and storage and transfer.
If you need more power check out the AX line of dedicated servers: https://www.hetzner.com/dedicated-rootserver/matrix-ax/
8 threads at 3.4 GHz, 8MB cache. Seems fine, depending on your use case.
6 replies →
For the 5 api requests a second most projects will get, it'll probably do.
> Except 40€ a month is extremely poor value for this CPU that's more than a decade old.
This is a rather baffling opinion to have. All cloud providers charge far more for a virtualized instance running on God knows what hardware. You are faced with a deal where you can run your software on bare metal, and you complain about... About what exactly?
Except you're getting a couple of disks, many GB of RAM, and some on-site 24/7 support, limitless network traffic, and your electricity bill.
Not too bad considering.
Excuse me, but if the difference between 10 EUR per month and 14 eur per month is going to kill your startup, you probably shouldn't try to start it. Might be time to think about using and creating less memory-hungry software.
Actually I disagree. I've killed projects because I've run out of time for them and didn't like them costing me £50 a month. If I'd been able to keep them going at £10 a month, I might have kept them going until I could get back to them. Sometimes startups fail just because the owners get distracted by life, and the project just needs more time.
Hetzner should not be compared to AWS or GCP for pricing. It should be compared to Vultr, Linode or DigitalOcean.
Of which DigitalOcean is running very outdated prices at 4USD for 512mb ram.
The cheapest Kimsufi dedicated server with 32GB ram is $11.10/mo.
1) this reads like it's posted by an LLM
2) why could they not just up the prices for new deployments, like they did with their dedicated servers? I think that would be fairer to existing customers
If you have a company, I can recommend leaseweb for cheap hosting. I host my personal stuff like my email and my ente.io instance there. They are cheaper than Hetzner (already before the new price increase) if you don't need managed k8s.
> Same kit I bought 8 months ago for 90 EUR is now 400+. That's not normal market dynamics.
That's exactly normal market dynamics during acute shortage. Remember 2020 when filtering face masks went up in price 10-100x?
On the plus siide, we all get to learn how few new computers are needed rather than chase number goes up.
> That's not normal market dynamics.
It is, in fact, normal market dynamics.
A single company that never made a profit outbiding the entire world is normal?
Yes. In fact that's not just normal, that's very frequent in market.
6 replies →
It's normal market dynamics for markets dominated by quasi-monopolies, which is why regulation should have prevented the existence of such markets.
I don't buy that it's because of the monopoly. TSMC has been starting a fab for close to 5 years.
It doesn't matter how many companies are in this market, it takes a real amount of time to add capacity.
Maybe not normal market dynamics, but typical human behavior: https://en.wikipedia.org/wiki/Vincent_Kosuga#Cornering_the_o...
Warning: another bot - check their history.
The reason Hetzner was cheap was bad latency and Arbor.
[flagged]
When I have looked on Newegg and on Amazon USA last month, I have seen even greater prices than here in Europe, by 30% to 40% greater, which is reversed from previous years, when computers and computer-related components were cheaper in USA than in Europe.
So I think that the victims are all the computer users of the entire world, with the exception of a negligible number of humans tied to the AI companies. Moreover, the US victims appear to be hit by the price hikes even more than in other countries, at least for now.
[flagged]
Man alive, stop being performatively dense. Getting to pay "the tax" in this context is just a colloquialism equating getting to pay "the burden".
That doesn’t make sense either. How is it a burden to pay the market price for something?
2 replies →
Everyone knows what they mean, it's obviously not a real "tax".