>drops the positional encodings after pretraining,

I thought with NoPE schemes you don't even need it during pretraining

https://arxiv.org/abs/2305.19466

https://sebastianraschka.com/llm-architecture-gallery/nope/

Given the causal mask and the way attention works LLMs can implicitly propagate a signal for ordering