I don't think anyone says it can't. Isn't that basically what ChatGPT does to interact with DALL-E?
I've been curious if DALL-E has been truly been mixed in w/ ChatGPT as a single model using a Mixture of Experts (MoE)- type learning gate to train them all together.