David Patterson is such a legend! From RAID to RISC and one of the best books in computer architecture, he's on my personal hall of fame.
Several years ago I was at one of the Berkley AMP Lab retreats at Asilomar, and as I was hanging out, I couldn't figure how I know this person in front of me, until an hour later when I saw his name during a panel :)).
It was always the network. And David Patterson, after RISC, started working on iRAM, that was tackling a related problem.
NVIDIA bought Mellanox/Infiniband, but Google has historically excelled at networking, and the TPU seems to be designed to scale out in the best possible way.
> To address these challenges, we highlight four architecture research opportunities: High Bandwidth Flash for 10X memory capacity with HBM-like bandwidth; Processing-Near-Memory and 3D memory-logic stacking for high memory bandwidth; and low-latency interconnect to speedup communication.
HBF is about having many dozens or hundreds of channels of flash memory. The idea of having Processing Near HBF, spread out, perhaps in mixed 3d design, would be not at all surprising to me. One of the main challenges for HBF is building improved vias, improved stacking, and if that tech advanced the idea of more mixed NAND and compute layers rather than just NAND stacks perhaps opens up too.
Weird to see no mention in this paper of persistent memory technologies beyond NAND flash. Some of them, like ReRAM, also enable compute-in-memory which the authors regard as quite important.
Why not, instead of passing the entire model through a processor and running it on every bit of data, pass the data (which is much smaller) through the model? As in, have compute and memory together in the silicon. Then you only need to shuffle the data itself around (perhaps by broadcast) rather than the entire model. That seems like it would use a LOT less energy.
Or is it not possible to make the algorithms parallel to this degree?
Edit: apparently this is called "compute-in-memory"
This is done that way at the GPU layer of abstraction - generally (with some exceptions!) the model lives in GPU vram, and you stream the data batch by batch through the model.
The problem is that for larger models the model barely fits in VRAM, so it definitely doesn't fit in cache.
Dataflow processors like cerebras do stream the data through the model (for smaller models at least, or if they can have smaller portions of models) - each little core has local memory and you move the data to where it needs to go. To achieve this though, Cerebras has 96GB of what is basically L1 cache among its cores, which is... a lot of SRAM.
Designing a concept sustainable RAM product and in working around multiplexing scaling challenges I somewhat accidentally developed a potential solution for hosting already-trained LLMs with very low energy and hardware in carbon and lignin;
> You have effectively designed a Diffractive Deep Neural Network (D^2NN) that doubles as a storage device.
Mode Division Multiplexing (MDM) via OAM Solitons potentially with gratings designed with Inverse Design of a Transition Map to be lasered possibly with a Galvo Laser. This would be a very low power way to run LLMs; on a lasered substrate
Frontier models are now much bigger than an individual query, hence batching, MoE, etc. So this idea, while very plausible, has economic constraints, you'd need vast amounts of memory.
Yes, this is the #2 direction recommended by the paper. Do you have arguments re "Table 4 lists why PNM is better than PIM for LLM inference, despite weaknesses in
bandwidth and power" ?
There are advantages, I suppose it comes down to economics and which of the advantages/disadvantages are greater. Probably if PIM was to ever catch on, it'd start off in mobile devices where energy efficiency is a high priority. Still might be impractical though.
David Patterson is such a legend! From RAID to RISC and one of the best books in computer architecture, he's on my personal hall of fame.
Several years ago I was at one of the Berkley AMP Lab retreats at Asilomar, and as I was hanging out, I couldn't figure how I know this person in front of me, until an hour later when I saw his name during a panel :)).
It was always the network. And David Patterson, after RISC, started working on iRAM, that was tackling a related problem.
NVIDIA bought Mellanox/Infiniband, but Google has historically excelled at networking, and the TPU seems to be designed to scale out in the best possible way.
> To address these challenges, we highlight four architecture research opportunities: High Bandwidth Flash for 10X memory capacity with HBM-like bandwidth; Processing-Near-Memory and 3D memory-logic stacking for high memory bandwidth; and low-latency interconnect to speedup communication.
High Bandwidth Flash (HBF) got submitted 6 hours ago! It's a great article, fantastic coverage of a wide section of the rapidly moving industry. https://blocksandfiles.com/2026/01/19/a-window-into-hbf-prog...
HBF is about having many dozens or hundreds of channels of flash memory. The idea of having Processing Near HBF, spread out, perhaps in mixed 3d design, would be not at all surprising to me. One of the main challenges for HBF is building improved vias, improved stacking, and if that tech advanced the idea of more mixed NAND and compute layers rather than just NAND stacks perhaps opens up too.
This is all really exciting possible next steps.
Why is persistence such a big thing here? Non-flash memory just needs a tiny bit of power to keep its data. I don't see the revolutionary usecase.
Density is the key here, not persistence.
5 replies →
Weird to see no mention in this paper of persistent memory technologies beyond NAND flash. Some of them, like ReRAM, also enable compute-in-memory which the authors regard as quite important.
Why not, instead of passing the entire model through a processor and running it on every bit of data, pass the data (which is much smaller) through the model? As in, have compute and memory together in the silicon. Then you only need to shuffle the data itself around (perhaps by broadcast) rather than the entire model. That seems like it would use a LOT less energy.
Or is it not possible to make the algorithms parallel to this degree?
Edit: apparently this is called "compute-in-memory"
This is done that way at the GPU layer of abstraction - generally (with some exceptions!) the model lives in GPU vram, and you stream the data batch by batch through the model.
The problem is that for larger models the model barely fits in VRAM, so it definitely doesn't fit in cache.
Dataflow processors like cerebras do stream the data through the model (for smaller models at least, or if they can have smaller portions of models) - each little core has local memory and you move the data to where it needs to go. To achieve this though, Cerebras has 96GB of what is basically L1 cache among its cores, which is... a lot of SRAM.
Designing a concept sustainable RAM product and in working around multiplexing scaling challenges I somewhat accidentally developed a potential solution for hosting already-trained LLMs with very low energy and hardware in carbon and lignin;
> You have effectively designed a Diffractive Deep Neural Network (D^2NN) that doubles as a storage device.
Mode Division Multiplexing (MDM) via OAM Solitons potentially with gratings designed with Inverse Design of a Transition Map to be lasered possibly with a Galvo Laser. This would be a very low power way to run LLMs; on a lasered substrate
In-memory processing: https://en.wikipedia.org/wiki/In-memory_processing
Computational RAM: https://en.wikipedia.org/wiki/Computational_RAM
Frontier models are now much bigger than an individual query, hence batching, MoE, etc. So this idea, while very plausible, has economic constraints, you'd need vast amounts of memory.
Yes, this is the #2 direction recommended by the paper. Do you have arguments re "Table 4 lists why PNM is better than PIM for LLM inference, despite weaknesses in bandwidth and power" ?
There are advantages, I suppose it comes down to economics and which of the advantages/disadvantages are greater. Probably if PIM was to ever catch on, it'd start off in mobile devices where energy efficiency is a high priority. Still might be impractical though.
Related too https://www.sdxcentral.com/news/ai-inference-crisis-google-e...
Yup, reads like the executive summary (in a good way).
Can’t we credit the first author in the title too? Come on.
The current title uses 79 characters of 80 character budget:
HN mods can revert the title to the original headline, without any author.
No we can't, that would be a crime against royalty :)
That appendix of memory prices looks interesting, but misses the recent trend.