• Earlier SD versions would often generate images where the head or feet of the subject was cropped out of frame. This was because random cropping was applied to its training data as data augmentation, so it learned to make images that looked randomly cropped — not ideal! To fix this issue, they still used random cropping during training, but also gave the crop coordinates to the model so that it would know it was training on a cropped image. Then, they set those crop coordinates to 0 at test-time, and the model keeps the subject centered! They also did a similar thing with the pixel dimensions of the image, so that the model can learn to operate at different "DPI" ranges.
• They're using a two-stage model instead of a single monolithic model. They have one model trained to get the image "most of the way there" and second model to take the output of the first and refine it, fixing textures and small details. Sort of mixture-of-experts-y. It makes sense that different "skillsets" would be required for the different stages of the denoising process, so it's reasonable to train separate models for each of the stages. Raises the question of whether unrolling the process further might yield more improvments — maybe a 3- or 4-stage model next?
• Maybe I missed it, but I don't see in the paper whether the images they're showing come from the raw model or the RLHF-tuned variant. SDXL has been available for DreamStudio users to play with since April, and Emad indicated that the reason for this was to collect tons of human preference data. He also said that when the full SDXL 1.0 release happens later this month, both RLHF'd and non-RLHF'd variants of the weights will be available for download. I look forward to seeing detailed comparisons between the two.
• They removed the lowest-resolution level of the U-Net — the 8x downsample block — which makes sense to me. I don't think there's really that much benefit from wasting flops on a tiny 4x4 or 8x8 latent tbh. Also thought it was interesting that they got rid of the cross-attention on the highest-resolution level of the U-Net.