What's worse is that this is probably going to get worse. My angel investment group is getting inundated with pitches that amount to building an RX-6000 with 96GB of RAM and installing a local model to do "thing X".
So even if the OpenAI's of the world stop trying to use up all the RAM, you're going to have thousands of start-ups pushing local models.
If Apple can't pull it off with their supply chain weight they can throw around, what is that thing going to be priced at? Microsoft/Nvidia are either going to be subsidizing it or it's gotta be close to $8,000+ at launch.
Why would anyone need that much RAM in a laptop?
Apple would be in a much stronger spot right now if they didn't pretend like eGPUs were inconceivable black magic that Macs are incompatible with.
In particular though, the fatal bottleneck is the weakness of the iGPU. Filling a KV cache on a 100gb+ model could take a few minutes, or even hours if you're trying to restore a 256k-to-1m token session.
This is already a thing
In general LLMs are bottlenecked by memory bandwidth rather than raw compute power.
And once the context gets large, it slows down.
"up to" 150, is doing a lot of work there.
But it's usable, fully local, fully private, and has no subscriptions and no operating costs other than electricity.