Sounds even worse (and I'm generally not a fan of speech to text interfaces, as typing is a lot quicker and easier on the brain); imagine a co-worker or your wife steps in and asks a question, your "Yes" turns into an answer to your coding agent.
But I don’t think you have it always listening, you probably tab the space bar to talk and send a prompt, walkie-talkie style, push to talk
It depends what kind of work you are doing. It's certainly not quicker for general discourse, even if you are a fast typer.