Potentially there would be a way for both sides to recognize that the other side is a machine and they could switch to binary communication.
I can give you 010101010101101011111 to a machine all I want, if they don't know how it's formatted, it's useless.
Conversational English is a format.
And as you said, format isn't enough. You need semantics.
If two applications know enough about the other side to know how to formulate their voice queries, they know at least enough to exchange those same queries as text, and skip the stupidly wasteful text->speech->text process.
(And if world wouldn't be so full of adversarial practices driving engineering stupidity, the developers would agree on an efficient binary format beforehand.)
https://www.youtube.com/watch?v=vvr9AMWEU-c
In other words, if M2M handshake works, switch away from voice.