How many buffers?
If you hang on to 10 frames of RGB plus motion vectors at 1080p, and have 4 4K buffers where you're messing with output, that's 6MB * 2 * 10 + 25MB * 4 = 220MB of video memory. That's barely anything. And that's an extra-high estimate of the resolutions that would be involved in a switch game.