Hacker News Reader

Attention is basically routing, these other routing schemes put a less fine-grained choice for the model, which potentially makes it easier to train

View on HN

How is attention basically routing?

It routes values based on linear combinations taken from the attention map.

But all of those values are created using an MLP with the same parameters, so there is no routing to different parameters.

You have to look at it as a sequence of time steps which can interact. You can implement this interaction in many ways, such as transformer, mamba, rwkv or mlp-mixer. But the purpose is always to allow communication across time.

You use three distinct linear projections, one for queries, one for keys and one for values. From Q and K you compute the attention matrix A, and using A you construct linear combinations from V. But depending on A, for example for a token V_i there might be input from two other tokens, V_j or V_k, so information is moved between the tokens.

Think of it like an edge flow matrix

That doesn't clarify it for me. The same parameters are being used for every layer for every token. Yes, there is this differentiable lookup in attention like in MoE - but routing is about more than just differentiable lookup, it is about selecting on parameters not state.

Reply on news.ycombinator.com