Hacker News Reader

To clarify, I'm talking about the original formulation of the problem, where positional encodings are crucial, because the article is talking about the original formulation of the problem.

But these days, there are many more options. They all maintain this positional information in one way or another. The one you cite (as I understand it) drops the positional encodings after pretraining, and keeps (e.g.) the causal mask.

View on HN

>drops the positional encodings after pretraining,

I thought with NoPE schemes you don't even need it during pretraining

https://arxiv.org/abs/2305.19466

https://sebastianraschka.com/llm-architecture-gallery/nope/

Given the causal mask and the way attention works LLMs can implicitly propagate a signal for ordering

You could be right, I have not kept up with the literature very closely! I'm just trying to help people understand that this article does not capture something very important about the Vaswani et al formulation of the transformer architecture. :)

Reply on news.ycombinator.com