These models work on a "latent" representation of the image, instead of working on the pixels directly. The model essentially learns a compression algorithm which allows it to process images most efficiently according to its loss function. The latent space might not represent certain types of shapes very well, as is the case with text characters.

Models like DeepFloyd IF work in pixel space, without learning a latent space. This means they can capture much more detail, but require more computational power. This is why these types of models usually rely on creating a smaller image and then using a separate model to upscale the image.