I don't look at this charitably. GPUs have several functions:
1) Driving monitors. If I want a dozen word processor windows open at the same time, I don't need a thousand CUDA cores, but I do need enough ports
2) Acceleration. This costs more money relative to CPUs today. Most people don't do much that requires this (although this might change with SD, LLMs, etc.)
3) Offloading bus load. Things like screen refresh, for an iGPU, requires constant access to main memory. On a GPU, there is dedicated graphics memory. A server showing a lock screen has constant bus load with an iGPU, but not with a dedicated graphics card.
... and so on.
As a footnote, fundamentally, what I would prefer is an architecture where I have:
* A normal frame buffer (of the type common in late nineties $20 graphics cards);
* A dedicated accelerator (many NVidia cards for server use don't even have graphics ports; they're GPGPU-only); and
* some kind of high-speed interconnect
At this point, I'm not sure there's much connection between CUDA cores and driving a graphics port. Driving a 4k display at 60 frames per second is about 1 GBps (upper-case B), so it's not a lot of interconnect in 2020-terms for a framebuffer and an accelerator to talk to each other.
Framebuffers ought to be simple and generic. There's no reason why GPU driver issues should introduce security or stability issues, or why I shouldn't be able to do multihead with different brands.