Translation is just a special case of general conversation. Just as LLMs can do translation just by asking, we ought to train a general speech-to-speech model for conversation, then we can simply ask it to translate for us.
AudioLM was developed more than a year ago and was an "audio continuation" model which is basically what you describe. However it was not used for conversational purposes afaik. Maybe it just needed some RLHF tuning?