Or upscale every 4th frame for consistency. Upscaling in between frames should be much easier.