"it's off while we're in the house." Are you sure about that?
If you mean "it might be listening! How do you know!" in the generic, fear-y sense then we can have that conversation about any network connected device you care to mention.
Agreed in that all devices are susceptible to 'listening' at any time without your direction or knowledge.