My favorite anecdote for showing how having a text encoder that actually understands the world is important to image generation is when querying for "barack obama" on a model trained on a dataset that has never seen Barack Obama the model somehow generates images of random black men wearing suits[1]. This is, in my non-expert opinion, a clear indication that the model's knowledge of the world is leaking through to the image generator. So if my understanding is right, as long as a concept can be represented properly in the text embeddings of a model, the image generation will be able to use that.
If my anecdote doesn't convince you, consider that one of google's findings on the Imagen paper was that increasing the size of the text encoder had a much bigger effect on not only the quality of the image, but also have the image follow the prompt correctly, including having the image generator being able to spell words.
I think the next big step in the text to image generation field, aside from the current efforts to optimize the diffusion models, will be to train an efficient text encoder that can generate high quality embeddings.
[1] Results of querying "barack obama" to an early version of cene555's imagen reproduction effort. https://i.imgur.com/oUo3QdF.png