If that was the case, nothing. But we have enough evidence to believe it's not. While having the device listen, even though it's been explicitly configured not to, is a bit more of a stretch.
>Are you just worried about accidental activation?
That could be an inconvenience too, but I'm more concerned with malicious intent.