I think that's exactly why they all have trigger words, "Ok Google", "Xbox", "Alexa". Makes it at least a little less likely.
The way I think they work is they constantly record 2 seconds of audio, and look only for peak "hard" sounds. Like K in Ok, and X in Alexa. If they find a peak they search back to see if the word was the one they are looking for, but otherwise the CPU is mostly sleeping.