FSR, XeSS, and DLSS negate this issue entirely actually.
See this DLSS diagram from nvidia for example: https://www.nvidia.com/content/dam/en-zz/Solutions/geforce/n...
If you hang on to 10 frames of RGB plus motion vectors at 1080p, and have 4 4K buffers where you're messing with output, that's 6MB * 2 * 10 + 25MB * 4 = 220MB of video memory. That's barely anything. And that's an extra-high estimate of the resolutions that would be involved in a switch game.
But we're talking about a computer with several extra gigs lying around running the same game. And specifically that number is for 4K output; anything less and you're under 100MB.