Are you using HDMI-CEC? You shouldn't be touching your TV remote at all. You shouldn't need to set a default HDMI input. All you have to do is press any button on the AppleTV remote and it will turn on the TV with the correct input device selected. I have a 2 year old Sony TV with GoogleTV and it works fine. I can go directly to a PS5 and Switch by just turning on the respective controller.