Most tvs have multiple HDMI inputs, why not have your Windows PC for gaming and a Linux PC with Kodi or whatever you like.
I personally just run mpv on my Linux box plugged into my TV and connected to a DLNA media server in my home and use my Android phone as a remote with KDEConnect, I'm not a big app person. I mostly watch YouTube videos my media server auto downloads when they are available from creators I like. For my family I have Netflix.
You're right it's not as nice as using a TV remote though! I bet Kodi or Plex, etc. would be more navigable with a remote. I wonder if you can get a Bluetooth remote that connects to a PC...