That’s fair, but again, a smartphone fits that description too and nobody is talking about how you’d have to be crazy to let one in your house.
It seems to me that smartphones are significantly worse. You can unplug your smart speaker or kick it off your WiFi and be confident that it can no longer hear you. Smartphones have batteries and cellular data connections. Also they’re usually with you at all times, even when you’re away from home.