There has got to be a number of fascinating tricks that they're using to support context lengths that long. Shame it's all closed-source.