Have you tried turning “match frame rate” on?
That’s probably the ticket for it though. I find the lack of audio pass through to be the more annoying piece though. I have a machine that costs significantly more than the Apple TV and has knowledge of all of the attached speakers to do that decoding.
That‘s the correct behavior though and not specific to the Apple TV. Any device that actually does switch and match the content’s frame rate will cause the output display to resync/adjust with a short black screen. The alternative of not adjusting frame rate is much worse and it’s such an underrated problem in video playback in my opinion. The Apple TV handles this better than most devices.