> squish the positional encodings for the keys down to position zero
I might be misunderstanding, but wouldn't this turn your instructions into a bag of words?
I might be misunderstanding, but wouldn't this turn your instructions into a bag of words?