Hacker News Reader

Comment by mapontosevenths | Hacker News Reader

> So z_t is a token in the input, and z_t+Δ is some later token.

I think you get what I'm saying, but when they compute the gate they do:

  h_t_new = alpha_t * h_t + (1 - alpha_t) * g_t

If I'm reading it right g_t = geometric_mix(z_t, z_(t+delta))

Which would mean that g_t includes t+Δ, which is a future token. It's leaking future tokens into the current math.

So it could just be a sign error and still be right in the code, or I could just be getting it wrong.

If I'm not though, and given that they got worse results than a standard transformer I really hope it's fixed in the code. If they do worse even when knowing the future it's bad news for the idea.

View on HN

Fair point. If only the paper had a link to the code.

I found this[1] repo which tries to reproduce it, and its accompanying blog post[2] uses (z_t, z_t - delta), without further mention.

Another repo[3] trying to reproduce it also does the reverse from what I can see, so (current, past).

Interestingly the first repo got underwhelming results, while the second repo matched the paper roughly. Not sure if it's related to the Claude-usage or not in the first, though it notes a lot of details were left out of the paper which is true.

[1]: https://github.com/Infatoshi/grassmann-flows/

[2]: https://github.com/Infatoshi/grassmann-flows/blob/main/blog....

[3]: https://github.com/kanenorman/grassmann

They advertised it as 10% to 15% worse than transformers

Not sure what the point of the paper's title is if the resulting model does not match the transformer.

Well they're quite explicit about that:

In summary, our results suggest that what we fundamentally need for strong sequence modeling is not attention as such, but a principled way for representations to move on the manifolds they inhabit. Grassmann flows provide one concrete instantiation of this idea, and we hope they will encourage further exploration of geometric alternatives to attention in the design of neural architectures.

They also note they kept the proposed model very simple to make it easier to understand (and presumably train):

In this paper we deliberately restricted ourselves to a minimal design—k = 2, no explicit global invariants—to keep the core idea clear. But we view “global invariants + local Grassmann flow” as a promising direction for future work on geometry-aware reasoning.

No my point is more general. The headline promises me that I don't need attention based transformers, then you see the benchmarks and they tell you they got 10% worse perplexity. How is that going to convince me that I don't need attention?

I mean if you look at e.g Mamba 3 [0] (see page 12 and 14 for benchmarks), the perplexity is roughly the same and the benchmark scores are roughly the same. If you tell me attention doesn't matter you have to convince me by showing that you can get the same performance with a variety of architectures. In short, I would have expected the paper to be a survey across model architectures that all perform roughly the same, which then would give us the impression "hey, they're all performing the same, maybe we don't need attention after all!", instead we get to be shown that their pet architecture is worse, which is the opposite of what the headline conveys.

If I gave a random ML researcher this paper, they would just walk away and work on the next transformer-mamba hybrid.

[0] https://arxiv.org/abs/2603.15569

Both of those reproductions are also b0rk3d. THey fix the sign flip, but then goof up other parts of the reproduction. Arledge (Infotashi), for example, has two residuals.

I still don't think it will work, but I'm also fairly confident that nobody has actually reproduced it well enough to know that for sure.

Reply on news.ycombinator.com