This is more of an engineering question than a physics one I think.
GPU dies are quite large compared to CPU, ballpark 2X to 3X, so I guess we could get a generation to generation and a half of transistor doubling out of increasing the size of the chips at least.
But the question is, do workloads benefit from all these transistors? Server workloads, I guess so, you can get some pretty big Xeons, or lots of transistors per package with AMD and multi-chip packaging. Consumer workloads, I think the issue is more that consumers don’t have these massively parallel workloads that servers and workstations have. Consumers have stated their preference by not paying for more than a handful of cores per chip.
The obvious way to get more transistors per core is fewer, bigger cores per chip. But changing the cores is a big engineering project. And yield issues are already a big problem, which would only become worse with more complex circuits. But until we end up with one core per chip, I think we have hit an engineering/economics problem rather than a physics one.
And the engineering/economics problem is a really big one, I mean AMD has had great success by going in the opposite direction; breaking things up further with multi-chip packages.