This is really not a good idea.

Modern stt is baking semantic EOT into the prediction. It makes sense. Conversation has many potential ends, and you have to deal with misprediction of current words that may signal completion, but mel spectrograms contain a ton of information as well.

In your version, we’d have to deal with partial transcriptions and a system that is not natively audio aware. It would also be much more expensive.