Hacker News Reader

Comment by schopra909 | Hacker News Reader

That sounds like an interesting idea!

Can you confirm I'm understanding correctly?

1) Linear unpatchify as usual to go from hidden states to pixel space

2) Attention within a local window (e.g. 3x3, 5x5) to "blend" pixel space data and come up with a better image (as an alternative to MLP or Convolution)

And follow up questions:

1) How do you handle boundaries between your "attention windows"? Do you move the window just like a convolution does or are the "attention windows" all mutually exclusive from one another?

2) How much faster/slower is this operation vs. a linear layer + MLP?

View on HN

1) yes, but with more than channels than 3 (and no correspondence to color, its "pixel space" in the sense of position, not value)

1D grid example for clarity (obviously meant to be done in 2D though)

So embed -> [N dim, N dim, N dim ... , N dim] instead of embed -> [RGB, RGB, RGB ... , RGB]

2) / [followup 1)] Stride of 1, so we place a window at each pixel (so lots of overlapping)

Also I wouldn't phrase it as "come up with a better image", the point is to give less spatial decoding pressure to the patch tokens so that they can almost completely focus on feature learning instead. There is no reason to have the model learn spatial decoding when the structure prior of images is comically strong (especially compared to text), its a waste of training time and parameters

[followup 2)] I haven't measured but it was passable is all I can say (my experiments setup are horrendous right now lol)

(Also don't mind the phrasing, I just wanted to be 100% clear)

I appreciate the clarifications!

If we have extra compute lying around in the coming weeks, we’ll try this out and report back. It’s a good idea :)

I loved to hear! Just ping me on twitter (same account I sent the link with)

I'm surprised you haven't heard about Natten, it's a few years old now: https://natten.org

It's the equivalent of sliding window attention, a common LLM optimisation. Global attention may be O(n2), but you can avoid that bottleneck by using local attention, even throughout the whole model. This could give you another efficiency boost, or enable you to process larger resolutions with linear instead of quadratic compute scaling.

Reply on news.ycombinator.com