Comment by Buttons840 | Hacker News Reader

Comment by Buttons840 | Hacker News Reader

If I want to do sequence modelling, let's say, predict the 9th element of the following sequence: [1, 2, 3, 4, 5, 6, 7, 8, 9] -- that is, I know the 8 most recent tokens as my "context", and I want to predict the next, 9 in this case --

Can someone explain to me why a transformer or RNN is better at this than a simple linear layer with an equivalent number of parameters? A linear layer can receive the context [1, 2, 3, 4, 5, 6, 7, 8], properly one-hot encoded / embedded, etc, and predict the next sequence. Can a linear layer do just as well as a transformer? This setup allows linear layers to predict sequences with an arbitrary context size, so why so much hype about transformers and RNNs and other sequence focused architectures?

Perhaps the difference is that given the same number of parameters, the transformer uses those parameters to perform easy computations whereas the linear layer just does one gigantic matrix multiplication which isn't very efficient?

Yea, this paper says that the embarrassingly simple linear is better than transformers for time-series:

https://arxiv.org/abs/2205.13504

from the abstract:

"Recently, there has been a surge of Transformer-based solutions for the long-term time series forecasting (LTSF) task. Despite the growing performance over the past few years, we question the validity of this line of research in this work. Specifically, Transformers is arguably the most successful solution to extract the semantic correlations among the elements in a long sequence. However, in time series modeling, we are to extract the temporal relations in an ordered set of continuous points. While employing positional encoding and using tokens to embed sub-series in Transformers facilitate preserving some ordering information, the nature of the permutation-invariant self-attention mechanism inevitably results in temporal information loss..."

The big difference is that the transformer is (approximately) permutation equivariant, which makes a massive difference in generalization and training speed.

I see, so [1, 2, 3, 4, 5, 6, 7, 8] is more similar to [5, 3, 1, 7, 8, 2, 4, 6] than with a linear layer? That's what you mean by permutation equivariant?

I understand each context input is embedded with its position, but I suppose the transformer can learn to ignore the position and just look at the context as an unordered set?

What makes it approximately permutation equivariant (vs entirely)? As I understand things, if the order is jumbled, the attention matrix does get its rows and cols permuted in the way you'd expect so I'd have thought they'd be entirely permutation equivariant.

Inputs have position encoding in them.

Reply on news.ycombinator.com