And does DALL·E just receive the image as-is with its description, or does it get more info (e.g. "this part of the image is the dog")?
And does DALL·E just receive the image as-is with its description, or does it get more info (e.g. "this part of the image is the dog")?
Web crawling. (https://arxiv.org/pdf/2103.00020.pdf 2.2)
> Are there people out there spending thousands of hours just tagging or writing descriptions of training images?
"Yes", Google Image Search, Wikipedia captions, Danbooru are all that in some sense.
The biases should be obvious if you think about it for a minute and CLIP has even more biases since they removed all kinds of NSFW content, etc.
> And does DALL·E just receive the image as-is with its description, or does it get more info (e.g. "this part of the image is the dog")?
Only that, but segmentation is a big research area too and it looks like FB has a reproduction with some of that mixed in.
https://www.gwern.net/Danbooru2021
It is a partially (highly) NSFW dataset though, which is probably the only way to get so much accurate volunteer tagging.
Also, if you want to just discuss the general concept of boorus (there are many beyond Danbooru), it'd probably be better to invoke Safebooru https://safebooru.org/ which is what it sounds like.
And the DALLE architecture can also handle masking where you initialize only part of the latent space with noise and initialize other parts with a starting image. The video on their website shows examples of that to replace a pet on a chair.
You're right, I forgot to mention them. Their metadata is great, and most importantly photos have their licenses tagged and many of them are CC0 (including mine).
Are content and content tags that great though? I don't content tag my own photos there, and when I've tried to comparison shop cameras all the popular images are over edited /r/shittyhdr art…
Metadata seems really interesting though. You'd think a visual search AI would want to know the white balance EXIF tags on a photo, so it knows if a yellow object is actually yellow or just under a streetlight.