Yarn-Mistral-7B-128k
huggingface.co
huggingface.co
One critique of the graph though... I measured perplexity of MistralLite myself, and its good out to 32K, but in the graph it drops off where the base model drops: https://huggingface.co/SciPhi/SciPhi-Mistral-7B-32k/discussi...
I'm not sure about that specifically (I am away from my desktop at the moment), but my 24GB card can handle Yi 34B 200K out to 50K+, depending on how much its quantized.
I would bet you can do 50K+ on 12GB with Mistral.
Ollama is very nice.
Llama.cpp is good for running on macs and for partial gpu offloading on non-macs.
There's also a table of memory usage depending on which file you use.
7B: 8GB (4GB min for model only)
13B: 12GB (8GB min)
34B: 28GB (18GB min)
70B: 60GB (38GB min)
Lower bpw quants and 8-bit cache size can reduce memory requirements / expand context, but if you're sticking to mainstream L1/L2 models, just go by that.7B fits comfortably in 6GB. 70B will squeeze, into 24GB, albeit with significant (but not catastrophic) quality loss.
It's not malicious, you can view the code yourself in the .py files: https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k/tre...