Yes it is quite expensive. The issue is memory bandwidth is a lot more constrained when youre routing to hundreds or thousands of cores instead of the handful you need to support on a CPU.
So the higher cost from higher VRAM comes from the implementation of higher bandwidth to accommodate more data flow because there are more cores versus CPU?