Integrated RAM with tensor processing seems like the end game for silicon that's actually efficient at running AI models.
Or integrated flash with TPUs? That would be awesome. Flash your model onto a model-processor and run it.
Or integrated flash with TPUs? That would be awesome. Flash your model onto a model-processor and run it.
The drastically different scaling trends for logic and SRAM on the same die are important, but I'm not sure how relevant it is to LLM inference. Cerebras has already shown that the amount of SRAM you can get out of a wafer is an absolute pittance compared to what frontier LLMs require; a multi-million dollar machine with ~40GB of SRAM is not a good solution for big LLMs no matter how fast it may be for small models. Denser forms of memory are almost mandatory for doing LLM inference on the largest models, and it seems pretty clear that the most cost-effective architectures we can build today for LLM inference would not be dominated by SRAM. You'll still have SRAM caches, but it doesn't make sense as the primary storage for the entire model's weights.