https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/im...
1) Retro, which is essentially attention over large databases, and fast as hell.
2) S4 Layers, explicitly designed for handling long dependencies.
These are orthogonal approaches to memory, and both very effective at what they do.
It was given a lot of tagged data: 600 million captioned images from LAION-5B. So if you want to know what it might support, you could try any one of the captions from those 600 million images.
The vocabulary is here: https://huggingface.co/openai/clip-vit-base-patch32/resolve/...
It is contextual though, so words in different orders mean different things.
Is there any reason it couldn't be character based, besides the (presumably very large) increase in resources needed to train and run inference? This is all way out of my league, but seems like you could get interesting results from this, since (by my caveman understanding) this hypothetical transformer could make some sense of words it had never seen before, so spelling variants or neologisms and such.
It's actually a byte-pair encoded (BPE is better than character encoding but can do the things you mentioned) list of things that includes words. You can find common English suffixes in it listed separately too.
GPT (and many other modern NLP models) use byte-pair encoding. Your summary of the benefits of this is correct - it can deal novel words much better.
Byte-pair encoding (BPE) is better than character encoding because it can deal with unicode (and emojis).
CLIP uses a BPE encoding of the vocabulary: The transformer operates on a lower-cased byte pair encoding (BPE) representation of the text with a 49,152 vocab size
So strictly this vocabulary is NOT (just) words, it is common sequences of byte pairs. You can see this if you examine the vocabulary - you'll find things like "tive" which isn't a word but is a very common English suffix.
The main difference is that coming up with a word SD doesn't know that's not contrived is really difficult. In an IF game, you are constantly guessing the correct word.
https://github.com/rom1504/img2dataset/blob/main/dataset_exa...
You probably would want to stop after getting the metadata, unless you have 240TB available for the images :)
More details and links to dataset explorers here: https://laion.ai/blog/laion-5b/
It's a bit less than 50k words, but that includes space-padded duplicates.
So in practice asking for art "in the style of <x>" is sort of limiting the denoiser to statistical pathways resembling other images captioned "in the style of <x>". At least, that's my understanding. Still trying to grok ML and diffusion models.
Having the art vocabulary down as well.
In effect, knowing what is present and how it’s tagged so you can « invoke » it more readily in the prompt-result.
Maybe I’m out of my depth. I know the corpus of tagged image used for training is enormous … but I still think that would help the user ( a prompt-crafter )
I will have to check myself.
I tryed with minor success to have stable diffusion draw lesser know non-us personality. It always kind of work, but the palette is limited. For instance I tried charles de gaule and you get something that look like him. But he's depicted talking on a radio or waving his arm around like a politician.
I tried to make him to grocery or play volley ball, it does not really work. While Michael jackson or Dennis Rodman get a way better treatment.
edit : that vocab file is smaller than I thought. "De Gaule" is not in there but neither Einstein or feiman? I think I missing something.
I Can find "obama", trump or Macron. Unclear about Michael Jackson. No beyonce or Denis Rodman. hmm weird, I had great result with all of them. Like.. recognizable details like tatoo or silly glasses.
It's just a very advanced madlibs engine based on a database of a billion of already known images.
If anything, this sort of "AI" only makes human ineffable creative experiences even more valuable, because without it there would be no training set and no "AI".
Do you have a source for that? Somehow I doubt there would be this much interest in a tool that literally just copies existing artwork.