Hacker News Reader

Reading that abstract, I'm not sure what is being claimed here. They say "the “state” in common SSMs is an illusion:" Is this a claim that the activations within a state space model contain no state or that it cannot contain an arbitrary state. The former seems trivially false and the latter seems completely uncontroversial.

View on HN

If you read the rest of the paper it's clear what they mean. SSMs are theoretically no more powerful than transformers, meaning without chain of thought they can't efficiently solve state-tracking problems, unlike RNNs: https://arxiv.org/abs/2207.00729 .

Not just efficiently, can't solve.

They can solve it if you keep adding layers to the transformer, it's just not efficient; you'd need exponentially more layers than a similarly sized RNN.

The difference to RNNs is that the state in SSMs is a linear combination of previous inputs, which makes it possible to parallelise training in various ways.

Gates and non-linearities in RNNs allow the state to be "any" function of previous inputs.

Reply on news.ycombinator.com