If physically disconnecting it is easy enough then I'd agree that is preferable. Only plug it in when required.

How about creating a firewall rule and then toggling that instead of physically disconnecting the Chromecast? I found a likely configuration: https://blog.g3rt.nl/allow-google-chromecast-host-firewall-i...