Comment by sroussey
9 hours ago
It’s SRAM that’s expensive, and more so as the nodes get smaller since the logic gets smaller but not the SRAM, which means it takes up a larger percent of die area over generations.
9 hours ago
It’s SRAM that’s expensive, and more so as the nodes get smaller since the logic gets smaller but not the SRAM, which means it takes up a larger percent of die area over generations.
In case you haven't been paying attention, DRAM and NAND are also pretty expensive these days, relative to where they were a year ago.
The drastically different scaling trends for logic and SRAM on the same die are important, but I'm not sure how relevant it is to LLM inference. Cerebras has already shown that the amount of SRAM you can get out of a wafer is an absolute pittance compared to what frontier LLMs require; a multi-million dollar machine with ~40GB of SRAM is not a good solution for big LLMs no matter how fast it may be for small models. Denser forms of memory are almost mandatory for doing LLM inference on the largest models, and it seems pretty clear that the most cost-effective architectures we can build today for LLM inference would not be dominated by SRAM. You'll still have SRAM caches, but it doesn't make sense as the primary storage for the entire model's weights.