There are people trying to frankenstain-merge Mistral and Whisper in a single multimodal model [1]. I wonder if this could improve the latency.

[1] : https://paul.mou.dev/posts/2023-12-31-listening-with-llm/