I’ll check it out. I tried the unsloth version but it couldn’t handle anything larger than an 8k context window unless I used a 2 or 3 bit quant

holy crap, thank you. Its actually running at 54 tok/s which is incredible for local inference.