Oh I have a TV that's "smart", but like you it's not connected at all to my WiFi. It's just a dumb terminal to my AppleTV. On levels of trust, I would put the AppleTV above my old Nvidia Shield and MUCH higher than my old Roku box.
My main gripe with smart devices is their sketchy software that may or may not have vulnerabilities in them and may or may not be diligently updated with security fixes by the company that makes them. If I can concentrate most things to one device...like the AppleTV, which gets updates all the time...then I can isolate problems. If I have to worry about a myriad of things throughout the house, that's something else.