Microphone access, especially if it's always-on to listen for the cue, is going to end up recording conversations, and those conversations will almost certainly involve people who are not the phone's owner and so could not have consented to having that phone record them.
(also there's the general problem of devices "helpfully" listening all the time for magic phrases like "Hey Siri" or "OK Google")