How about Mistral 7B? I saw this article recently:
https://wandb.ai/byyoung3/ml-news/reports/Fine-Tuning-Mistra...
https://wandb.ai/byyoung3/ml-news/reports/Fine-Tuning-Mistra...
What would be interesting is compare GPT4 at a certain task with a small model fine tuned for that task.
Even if each inference pass only runs part of the network, there's still a trillion learnable parameters there lol.