The primary source is the liked Twitter thread. I wonder how credible this source is. (I'm not familiar with the norm of ML community - They seem to be Twitter-heavy than other part of tech.)
The OP doesn’t explain anything. It just vaguely talks about a few things that might break when scaling context. But that means nothing.
Take for example sinusoidal embeddings they talk about. Of course it breaks for large contexts but in no one uses it. GPT uses learned positional embeddings so the entire section is irrelevant.
Copy this for pretty much everything else.
Being an expert in a field has never been this exhausting
The ALiBi stuff just makes sense to me. I don't understand why the Positional Sinusoidal Encoding was used initially. I assume there were good reasons for it but I haven't seen an explanation, (pointers to one appreciated).