Thats basically what Intel Falcon Shores is, and I am sure AMD is cooking up something similar behind the MI300.
We already have the ability to put far more memory in GPUs than they currently have, but we don't because it hurts throughput. I don't think technical advances will change the existence of a throughput/capacity tradeoff, but I do think it's an interesting idea that maybe DL models benefit from a different tradeoff than graphics applications. Personally I'd guess that's a larger factor here than incremental changes in capabilities, because the two types of memory are both improving at the same time. But who knows! Maybe the technologies will converge enough that it will no longer make sense to have two different variants, though I wouldn't bet on it.
Actual ML GPUs transitioned to HBM, but the capacity is relatively limited. I think Nvidia thought 96GB per GPU would be plenty when they were planning things out years ago, but the explosion of model sizes seems to have taken all the hardware manufacturers by surprise.
https://www.amazon.com/ATI-Rage-Video-Card-109-43200-10/dp/B...