My understanding is that this is already the case, see PaLM-E as one such example of a multimodal model.