This being a 3B model isn't remotely comparable to GPT4.
WizardCoder 34B and Phind 34B are the only models remotely comparable, and they are still slightly worse than GPT 3.5 (let alone GPT4).
WizardCoder 34B and Phind 34B are the only models remotely comparable, and they are still slightly worse than GPT 3.5 (let alone GPT4).
https://wandb.ai/byyoung3/ml-news/reports/Fine-Tuning-Mistra...
What would be interesting is compare GPT4 at a certain task with a small model fine tuned for that task.
Even if each inference pass only runs part of the network, there's still a trillion learnable parameters there lol.
So comparing it with big models I'd say it's good but might have limited usefulness
(you can probably go further with 3B with only code)