It's far, far easier in practice to use an unambiguous gesture like a button press on a phone than it is to use an unambiguous gesture with a smart speaker.
Most normal people could get by with a voice assistant on their phone that required manual prompting, but a voice assistant like Alexa that required you to walk across the room and push a button to use it would be useless -- at that point carrying your phone in your pocket would be a better UX experience.
How the products are designed and implemented in the real world matters: phones are designed to be in-reach of people, smart speakers are not. You can't just stick a mute button on a smart speaker and say it's the same UX as phones now.
> How many people that refuse to have a smart speaker also refuse to have a smart phone?
If somebody is avoiding a smart speaker out of privacy concerns, but they have an always-on voice assistant on their phone, the correct response to their concerns is not to tell them to stop worrying about smart speakers. The correct response is to educate them about their phone settings so they can change them and exercise more agency over their lives.