I'm a bit rusty on the details, but I think the general reason that GPUs have less memory than you want is because they are typically optimized for throughput, which limits capacity in a couple ways. One is higher power consumption, and another is low max distance between processors and memory.

We already have the ability to put far more memory in GPUs than they currently have, but we don't because it hurts throughput. I don't think technical advances will change the existence of a throughput/capacity tradeoff, but I do think it's an interesting idea that maybe DL models benefit from a different tradeoff than graphics applications. Personally I'd guess that's a larger factor here than incremental changes in capabilities, because the two types of memory are both improving at the same time. But who knows! Maybe the technologies will converge enough that it will no longer make sense to have two different variants, though I wouldn't bet on it.