I’ve heard rumbling about native context length. I don’t know too much about it, but is this natively 1M context length?
So even models like llama3 8b say they have a larger context, but they really don’t in practice. I have a hard time getting past 8k on 16gb vram (you can definitely set the context length higher, but the quality and speed degradation is obvious).
I’m curious how people are doing this on modest hardware.