Have you tried increasing the repetition penalty? Needing to do this is already a red flag, but I'm curious if it helps at all.

Running LLMs locally on outdated hardware, I've come to the realisation that sub-10b models are broadly speaking kind of stupid. They can answer simple questions, but can't really solve problems.

The effect is so strong that with an 8GB GPU a 27b model can sometimes finish the task earlier than a 9b model, despite residing largely in RAM. If it's a MoE model, then it isn't even a competition.