And despite the theoretically high memory bandwidth of recent EPYC CPUs, approximately nobody who can afford one is doing LLM inference on them.
22,096 karma · joined September 20, 2009
And despite the theoretically high memory bandwidth of recent EPYC CPUs, approximately nobody who can afford one is doing LLM inference on them.
And it seems more plausible to me that an ASIC architecture rather than GPUs would be able to best make use of something like wafer-bonded custom memory to approach the density of discrete DRAM while retaining the extremely high bandwidth that comes with arbitrarily wide interfaces and minimal PHYs.
Designing for extreme expandability comes with pretty steep tradeoffs.
I'm not sure there's any reason to expect a similar shift from LLMs. The hardware used for training doesn't dictate what hardware needs to be used for inference, and nobody's going to design an LLM architecture with an overt intention to make it better suited to GPUs and hard to target with ASICs.
It's still implausible, but thoroughly verifying that your TV isn't spying on you isn't trivial.
Back when Chrome was new, you could run Firefox with a decent selection of add-ons to get security benefits of similar magnitude through different mechanisms. The NoScript extension was at the forefront, providing a lot of functionality way beyond just blocking JavaScript per domain. And Firefox with extensions always had better privacy controls.
The drastically different scaling trends for logic and SRAM on the same die are important, but I'm not sure how relevant it is to LLM inference. Cerebras has already shown that the amount of SRAM you can get out of a wafer is an absolute pittance compared to what frontier LLMs require; a multi-million dollar machine with ~40GB of SRAM is not a good solution for big LLMs no matter how fast it may be for small models. Denser forms of memory are almost mandatory for doing LLM inference on the largest models, and it seems pretty clear that the most cost-effective architectures we can build today for LLM inference would not be dominated by SRAM. You'll still have SRAM caches, but it doesn't make sense as the primary storage for the entire model's weights.
The problem with your rambling and speculation is that you're not making any attempt to think about what conditions would be necessary for a technology to succeed in the market and gain widespread adoption across many market segments. I stand by my assertion that it is impossible for compute-in-memory chips to entirely replace commodity DRAM, especially in any market segment where the compute portion of those memory chips is not critically valuable. Any remotely plausible, useful compute-in-memory design will have significantly lower capacity and higher cost than pure commodity DRAM; this is not a low-overhead added feature. Any market segment where the compute capabilities of compute-in-memory were so marginally valuable that a non-trivial number of the customers wanted to disable the compute is a market segment where everybody would still be buying mostly ordinary DRAM.
The worst of Intel's processor core asymmetry was with Meteor Lake and Arrow Lake, where the two LP-E cores on the IO die had no L3 cache and really bad DRAM latency despite being directly adjacent to the DRAM controller, resulting in those two cores combined offering about as much performance as one of the E-cores on the CPU chiplet. And they used the same IO die across both generations, while updating the microarchitecture of the CPU chiplet, so things diverged even more. And because the performance of the LP-E cores was so bad, Windows wouldn't schedule any threads on them unless it was trying to operate exclusively on those LP-E cores with the CPU chiplet powered down entirely—but those two LP-E cores were still reported to software in the total thread count, so heavy multithreaded applications would launch two more threads than could actually be scheduled at once.
With Panther Lake, Intel still has three tiers of CPU cores, but the performance gap between E cores and LP-E cores is down to only about 10%.