On a related note, I was reading yesterday that apparently the real bottleneck for Chinese production of AI accelerators is HBM production, not processors or ASML equipment.
The lack of ASML EUV machines certainly hurts, and pushing DUV so hard results in abysmal yields of good chips, but you can compensate by running more wafers or making smaller chips, and the net result is that Huawei's Ascend production volume is limited by CXMT's HBM capacity not processor dies.
The problem is that HBM manufacture requires many steps (die thinning, via drilling, plating, alignment) where the equipment used by everyone else (Samsung, SK Hynix, Micron) is also blocked by sanctions, so the Chinese are having to develop all of this themselves too, which they have, but yields are currently low, even when using shorter HBM stacks.
It's not that there's a blocker. It's that it takes roughly 3x the manufacturing capacity to produce an HBM package at the same storage capacity as DRAM. We are sacrificing total bytes for bandwidth.
I don't fully understand the source of the "total bytes" constraint, but a major factor may be because HBM4 / HBM4E can only make use of the footprint directly above the processor/logic die (or in direct vicinity of its interconnect), while traditional DRAM can be placed further away where there's lots of real estate on the motherboard.
I gather a practical max ceiling today is a stack of 16 chips in height yielding 64GB?
These chips have a massive bus size of 2048 bits, instead of the 64 or 128 bits (dual channel) used by DDR5. That's what gives them their order-of-magnitude bandwidth speedup. But even though they technically pack in more capacity per square millimeter of motherboard, I gather they take up more space than older technologies once you account for the vias and interconnects to route all those signals.
How so? FEOL is pretty much the same, BEOL is almost the same save TSVs, the packaging tech is different and more advanced, but not exactly 1:1 comparable. Do TSVs really occupy 3x the area of DDR IO's?
Mainly bus width. Afaik HBM is like 1024 bits vs DDRs 64 so you need lots of transfers in parallel to saturate the bus, and CPUs kinda want 64 bytes of data as that's the size of a cache line ASAP. So you need a ton of in flight transfers which isn't a thing CPUs provide, maybe multicore workloads.
Buy the way you win with CPUs is with latency, and not bandwidth, which is why Apple M series actually uses DDR with lower latency because of the stacking.
I think people might prefer the lower idle power consumption from lpddr over the better bandwidth in hbm in battery powered stuff. That said right now the price is definitely preventing us from finding out.
The advantage of HBM over regular non-stacked DRAM is memory bandwidth, which also requires a super-wide memory bus - 2048 bits wide for HBM4. Compare that to the 128 bit wide bus of a modern CPU.
So to take advantage of it on the desktop, or anywhere else, you need that 2048 bit wide bus, and a processor capable of consuming 2-3 TB of data per second!
These are not normal requirements, other than for a GPU.
In one sense, nothing, in another, everything. It is DRAM, but the bandwidth requirements mean it’s paired to a processor, i.e. no DIMMs. Not 100% sure but things like MacBooks and the Framework tower, where you have fixed RAM for the device lifetime, have ~0 tradeoff.
The normal non-tech-savvy person already does this. They simply don't know that their ram is upgradeable or something else breaks first, before having to touch ram.
HBM4 and HBM4E DRAM are NOT the DDR4/5 that consumer markets need. Capacity allocation is leaning more to data center grade HBMs so less to produce dedicated DDR4/5 DRAMs. Supply demand will further drive up the consumer DRAM price! Note, the article mentions NO of new fabs is being constructed (all semiconductor manufacturers know that constructing more fabs means the boom/burst cycle will eventually kill them, so no one create more fabs) Perhaps the federal government need to step in here - the market doesn't fit the issue.
Depends on what proportion of Samsung's output is current HBM4 and HBM4E DRAM.
Everything in their statement can be true and it be a bad thing for consumers of non-HBM RAM.
"HBM capacity to expand to 250,000 wafers a month"
So let's say their current HBM capacity is 100k wafers/month (pure speculation/random number for illustration), and their total RAM capacity (including HBM( is 300k wafers/month, then non-HBM capacity reduces from 200k to 50k.
Part of the implication is that factories that could be producing consumer-facing DRAM like DDR5 would be retooled to produce HBM instead, leading to even less total consumer RAM production.
If Samsung is limited in the number of wafers they can process per month and they use more of those to produce HBM they necessarily have less of them left to make other products, like consumer DRAM.
During the Covid global chip shortage Intel announced new factories to address the production bottleneck, before the factories even started to be constructed, the shortage was long over and the projects were eventually canceled.
What if the AI companies start selling the hardware with their models?
I.e., you get a locked down device with access to their AI and maybe a way to run third party apps. Of course the developers of those apps will have to pay a percentage of their revenue.
That's never going to happen. By the time you can run current frontier models on your $10k desktop the frontier will have massively advanced and people will want those models instead.
Samsung is already building another fab (which was originally suspended due to low memory prices years ago). Hopefully when it goes live in a few years it's not all dedicated to HBM for AI.
About 2 years ago I bought some SDRAM or something like that, DDR4 or DDR5, don't recall offhand. A few months ago I looked at the price today, and it was over
3x as high. That's just insane. Governments need to do something against this abuse system that AI amplified here.
> Governments need to do something against this abuse system that AI amplified here.
Which governments, and what exactly would you want these governments to do? The demand is global. There are no levers a single government can pull to meaningfully influence the global demand without fully committing to an protectionist economic policy, in which case the U.S. doesn't have the facilities to magically pop up world class fabs overnight, and South Korea and Taiwan don't have the market demand that the U.S. generates to justify their investments in making these chips and China lacks the IP to be able to build anything comparable to Nvidia's silicon at the moment.
No one has all the cards and no one controls all the levers.
Governments have sued DRAM companies for price fixing and forming cartels before: https://en.wikipedia.org/wiki/DRAM_industry_price_fixing. They can absolutely step in and if there are evidence of price fixing or anti-competitive behaviours, can impose fines. The choice of whether to do that though is political.
No one has all the cards and no one controls all the levers.
I'm as fanatical a free-market fundamentalist as you'll ever meet, but if someone were to argue that government has a role in preventing bullshit like OpenAI's unilateral 40% attack on the entire DRAM market, backed by nothing but funny money, I would have a hard time coming up with defensible counterarguments.
On a related note, I was reading yesterday that apparently the real bottleneck for Chinese production of AI accelerators is HBM production, not processors or ASML equipment.
The lack of ASML EUV machines certainly hurts, and pushing DUV so hard results in abysmal yields of good chips, but you can compensate by running more wafers or making smaller chips, and the net result is that Huawei's Ascend production volume is limited by CXMT's HBM capacity not processor dies.
The problem is that HBM manufacture requires many steps (die thinning, via drilling, plating, alignment) where the equipment used by everyone else (Samsung, SK Hynix, Micron) is also blocked by sanctions, so the Chinese are having to develop all of this themselves too, which they have, but yields are currently low, even when using shorter HBM stacks.
Tokens per second is almost entirely memory bandwidth at inference time, training obviously needs more compute but you can add more chips for that.
China should invest in an analog inference chip. It's a hail mary but why not.
I read HBM yields are 25-30% (vs 80-90%) making them 3 to 5 times as expensive. They are 4-5 years behind, that probably means 1-2 in Chinese time.
Does that include yield from packaging?
What's the main blocker (other than the current inflated cost) for using HBM instead of DRAM as the primary memory for consumer electronics?
It's not that there's a blocker. It's that it takes roughly 3x the manufacturing capacity to produce an HBM package at the same storage capacity as DRAM. We are sacrificing total bytes for bandwidth.
I don't fully understand the source of the "total bytes" constraint, but a major factor may be because HBM4 / HBM4E can only make use of the footprint directly above the processor/logic die (or in direct vicinity of its interconnect), while traditional DRAM can be placed further away where there's lots of real estate on the motherboard.
I gather a practical max ceiling today is a stack of 16 chips in height yielding 64GB?
These chips have a massive bus size of 2048 bits, instead of the 64 or 128 bits (dual channel) used by DDR5. That's what gives them their order-of-magnitude bandwidth speedup. But even though they technically pack in more capacity per square millimeter of motherboard, I gather they take up more space than older technologies once you account for the vias and interconnects to route all those signals.
1 reply →
How so? FEOL is pretty much the same, BEOL is almost the same save TSVs, the packaging tech is different and more advanced, but not exactly 1:1 comparable. Do TSVs really occupy 3x the area of DDR IO's?
9 replies →
Mainly bus width. Afaik HBM is like 1024 bits vs DDRs 64 so you need lots of transfers in parallel to saturate the bus, and CPUs kinda want 64 bytes of data as that's the size of a cache line ASAP. So you need a ton of in flight transfers which isn't a thing CPUs provide, maybe multicore workloads.
Buy the way you win with CPUs is with latency, and not bandwidth, which is why Apple M series actually uses DDR with lower latency because of the stacking.
>> What's the main blocker (other than the current inflated cost) for using HBM instead of DRAM as the primary memory for consumer electronics?
HBM is meant to be integrated into the same package as the CPU, so no more DIMM sockets. It also has higher latency apparently.
I think people might prefer the lower idle power consumption from lpddr over the better bandwidth in hbm in battery powered stuff. That said right now the price is definitely preventing us from finding out.
Idle yes, but HBM energy consumption / memory operations seems to be a bit better than DRAM.
4 replies →
Why would you want/need to?
The advantage of HBM over regular non-stacked DRAM is memory bandwidth, which also requires a super-wide memory bus - 2048 bits wide for HBM4. Compare that to the 128 bit wide bus of a modern CPU.
So to take advantage of it on the desktop, or anywhere else, you need that 2048 bit wide bus, and a processor capable of consuming 2-3 TB of data per second!
These are not normal requirements, other than for a GPU.
SIMD (and especially the modern matrix extensions) can use as much bandwidth you can throw at it.
2 replies →
Right but if HBM memory is all that people want to produce, then building a CPU which can use it use it would be useful on it's own merits.
But in reality we also already have unified memory architecture systems, integrated graphics etc.
2 replies →
In one sense, nothing, in another, everything. It is DRAM, but the bandwidth requirements mean it’s paired to a processor, i.e. no DIMMs. Not 100% sure but things like MacBooks and the Framework tower, where you have fixed RAM for the device lifetime, have ~0 tradeoff.
HBM is a stack of DRAMs, so there is no “instead”.
The vias to enable stacking is a significant amount of the die area.
1 reply →
Nothing except CPU manufacturer choices. Mac laptops use it and they're consumer products.
People will have to get used to buying a fixed amount of RAM with their CPU but thats unlikely to be a problem.
Apple's "unified memory" marketing is so strong that even tech literate people seem to have this misconception!
They have managed to pull this sort of thing off many many times. https://en.wikipedia.org/wiki/Reality_distortion_field
17 replies →
MacBooks use regular soldered lpddr5(x) RAM. Same RAM as every other laptop manufacturer, they just use more lanes to achieve a higher bandwidth.
3 replies →
The normal non-tech-savvy person already does this. They simply don't know that their ram is upgradeable or something else breaks first, before having to touch ram.
1 reply →
> Mac laptops use it and they're consumer products
No, Mac laptops use LPDDR, currently LPDDR5X.
pretty sure its lpddr not hbm.
tl;dr; This is existing production being redirected to HBM. Which means consumer market is going to get a lot worse.
And the worst thing is that is the best possible strategy for them. It's essentially win-win for everyone but consumers.
- Buyer (AI) has crazy money, so will pay whatever
- Seller doesn't have to build anything new, as the buyer is willing to pay whatever
- Generate ridiculous profits from crazy money
- No oversupply risk in case of reversal
- Return from producing HBM to DDR5 in a single quarter if reversal does happen.
Hold on to your existing hardware, people, and be on lookout for your local deals.
A shame that this should if anything, lead to consumer DRAM prices getting even worse.
Why would more RAM supply lead to higher consumer prices?
HBM4 and HBM4E DRAM are NOT the DDR4/5 that consumer markets need. Capacity allocation is leaning more to data center grade HBMs so less to produce dedicated DDR4/5 DRAMs. Supply demand will further drive up the consumer DRAM price! Note, the article mentions NO of new fabs is being constructed (all semiconductor manufacturers know that constructing more fabs means the boom/burst cycle will eventually kill them, so no one create more fabs) Perhaps the federal government need to step in here - the market doesn't fit the issue.
4 replies →
Depends on what proportion of Samsung's output is current HBM4 and HBM4E DRAM.
Everything in their statement can be true and it be a bad thing for consumers of non-HBM RAM.
"HBM capacity to expand to 250,000 wafers a month"
So let's say their current HBM capacity is 100k wafers/month (pure speculation/random number for illustration), and their total RAM capacity (including HBM( is 300k wafers/month, then non-HBM capacity reduces from 200k to 50k.
4 replies →
Part of the implication is that factories that could be producing consumer-facing DRAM like DDR5 would be retooled to produce HBM instead, leading to even less total consumer RAM production.
Memory allocated for HBM is memory taken away from DDR production.
If Samsung is limited in the number of wafers they can process per month and they use more of those to produce HBM they necessarily have less of them left to make other products, like consumer DRAM.
Gamma squeezed by the AI business, that's why. The more you add the fuel, the further it will pump up.
Presumably DRAM production capacity gets reallocated to HBM, not sure.
Depends on how quickly they are sold.
Maybe this leads to device manufacturers using HBM memory instead.
Will that be enough for AI's hunger?
During the Covid global chip shortage Intel announced new factories to address the production bottleneck, before the factories even started to be constructed, the shortage was long over and the projects were eventually canceled.
It will be just in time for when AI will run very well on consumer hardware and the need for data centers will collapse.
What if the AI companies start selling the hardware with their models?
I.e., you get a locked down device with access to their AI and maybe a way to run third party apps. Of course the developers of those apps will have to pay a percentage of their revenue.
Sound familiar?
1 reply →
There will be demand for both
That's never going to happen. By the time you can run current frontier models on your $10k desktop the frontier will have massively advanced and people will want those models instead.
28 replies →
Or even more amusing in time for the bubble to burst, I hope the big RAM makers end up holding the whole bag for that. Greedy bastards.
8 replies →
Samsung is already building another fab (which was originally suspended due to low memory prices years ago). Hopefully when it goes live in a few years it's not all dedicated to HBM for AI.
Not even close.
Not until the industry gets severe indigestion, which doesn't seem that far off.
The beginning of the bubble pop?
how would you even derive that conclusion from the headline/article?
its ram for the types of systems ai needs in datacenters so...no?
Now make some DDR5.
HBM is DDR5 in a different package.
If it doesn’t fit it’s no use to me.
[flagged]
Why wouldn't it get posted here?
Because you are not familiar with the site.
About 2 years ago I bought some SDRAM or something like that, DDR4 or DDR5, don't recall offhand. A few months ago I looked at the price today, and it was over 3x as high. That's just insane. Governments need to do something against this abuse system that AI amplified here.
> Governments need to do something against this abuse system that AI amplified here.
Which governments, and what exactly would you want these governments to do? The demand is global. There are no levers a single government can pull to meaningfully influence the global demand without fully committing to an protectionist economic policy, in which case the U.S. doesn't have the facilities to magically pop up world class fabs overnight, and South Korea and Taiwan don't have the market demand that the U.S. generates to justify their investments in making these chips and China lacks the IP to be able to build anything comparable to Nvidia's silicon at the moment.
No one has all the cards and no one controls all the levers.
Governments have sued DRAM companies for price fixing and forming cartels before: https://en.wikipedia.org/wiki/DRAM_industry_price_fixing. They can absolutely step in and if there are evidence of price fixing or anti-competitive behaviours, can impose fines. The choice of whether to do that though is political.
5 replies →
No one has all the cards and no one controls all the levers.
I'm as fanatical a free-market fundamentalist as you'll ever meet, but if someone were to argue that government has a role in preventing bullshit like OpenAI's unilateral 40% attack on the entire DRAM market, backed by nothing but funny money, I would have a hard time coming up with defensible counterarguments.
6 replies →
How much memory does it take for the consumer market not to be totally screwed?
Can that share of production be allocated to consumers, and the AI fights over the rest?
What abuse? Responding to supply and demand fluctuations isn't abuse, it's the proper operation of the market.
Would if one industry gobbles up an important resource to the point that other consumers lose practical access to it, shouldn't there be limits?
7 replies →
Where is the additional supply resulting from the sustained increase in demand?
2 replies →
3x? I bought 128GB DDR5 SODIMMs for 300 eur. Now it's 4000 eur.
Damn you were lucky. I bought 96GB DDR5 last year for £600 UKP. They're now retailing for three times that.
Bastards.
[dead]