Comment by wtallis
6 hours ago
I wonder if we might eventually see stacking of NAND, logic, and DRAM dies together so each can still be fabbed on their specialized process. Getting all three on one wafer is extremely impractical, but having them all connected with TSVs or other bonding might be feasible, if the density of the respective dies works out right to give you the right ratio of storage for weights, working memory, and compute.
It’s SRAM that’s expensive, and more so as the nodes get smaller since the logic gets smaller but not the SRAM, which means it takes up a larger percent of die area over generations.
In case you haven't been paying attention, DRAM and NAND are also pretty expensive these days, relative to where they were a year ago.
The drastically different scaling trends for logic and SRAM on the same die are important, but I'm not sure how relevant it is to LLM inference. Cerebras has already shown that the amount of SRAM you can get out of a wafer is an absolute pittance compared to what frontier LLMs require; a multi-million dollar machine with ~40GB of SRAM is not a good solution for big LLMs no matter how fast it may be for small models. Denser forms of memory are almost mandatory for doing LLM inference on the largest models, and it seems pretty clear that the most cost-effective architectures we can build today for LLM inference would not be dominated by SRAM. You'll still have SRAM caches, but it doesn't make sense as the primary storage for the entire model's weights.
Isn't Intel effectively betting the farm on this right now? Other than stacking technology, what do they have that's even a little interesting?