What techniques do they actually use to achieve 100k? I assumed they load the document to a vector databases and then do some kind of views into that
(I think I’ve got to read up on how transformers actually work.)
Recurrent neural networks are bad when the recurrence is 100x long or more. You need long chains because with a token-at-a-time, that's what you need to process even one paragraph.
But if you use an RNN around a Transformed-based LLM, then you're adding +4K or +8K tokens per recurrence, not +1.
E.g.: GPT 4 32K would need just 4x RNN steps to reach 128K tokens!