Interesting... is it overkill to use a multimodal model like this on just audio? Could the model size be decreased dramatically, or does the multimodality help the accuracy and speed in some way?
The benefit is that you can take input in one "mode" and extract output in another; that is, the input is audio mode and the output is text mode. The intermediate representation is the "embedding".
Multimodality seems to just help with everything
Correctly transcribing requires context too, no?
I suppose it depends on the words in question... language knowledge and general world knowledge may not always both be necessary to transcribe accurately.