I Implemented Transformer in Transformer | Hacker News Reader

I Implemented Transformer in Transformer

github.com

37 pointsdagli10 comments

Transformers are basically CNNs with fixed-sized convolutions per layer (called heads in the lingo). Whereas CNNs have different-sized convolutions at each layer.

Instead of looking at things through many lenses, like a CNN, the Transformer cross-examines everything through the same lense (aka attention or focus).

Transformers-in-Transformers (TiTs) is, in effect, adding more CNN-like layers to each regular transformer layer.

How might one be able to use human rating data to inform transformers? Like, if the outputs are shown 3 at a time to users and the user chooses the best, how to create a feedback loop based on this labeled data? Or, if large numbers of output images/texts are individually rated on a scale of 0-10 by aesthetic beauty—-how might this be fed back to the transformer?

You could use this to create a heatmap of an image to identify regions that would be rated as "beautiful," at the pixel level.

You could then use that to inform a gan, iteratively modifying the image to whatever parameters you've trained the transformer to look for, gradually beautifying the image.

Instead of beauty, you could use this for a Clip type model, judging quality and accuracy of the output, using the model to guide the training of a Dall-E style model.

There are some obvious applications in trading, creating a model that incorporates multiple time frames and text feeds.

This paper is pretty cool!

Are you still in school or in uni now rishit?

Repo belongs to a high schooler. Respect !

Can someone explain this better for me and how it would be more useful than a regular transformer?

Attention is more nuanced, operating at different scales. A common flaw in deep fake faces is pupils that aren't quite round, so this type of model could be used to target errors like that. Or if you had a picture of a library bookshelf, the model can attend to the details of individual books, and also attend to the letters of their titles at a per book level.

Dangerous waters. If you implement another transformer in that transformer in your transformer, you could cause recursion and it could end up in what is called a 'voltage collapse' - sucking the entire universe within the recursion.

Tread lightly. With great power comes great responsibility.

A light-hearted joke, but it doesn't add to the conversation, so it isn't appropriate for hacker news.

Thank you for your carefree and refreshing response.

Reply on news.ycombinator.com