While I’m not necessarily in favor of this, a multimodal AI that has access to your location, vision inputs, etc could obtain much of this context. People already explore foreign countries by hobbling together these services.

Human translators do it in approximately real-time, see for example the United Nations. At most you might need to wait for the end of the sentence to translate if e.g. the verb/object ordering is different.