Diligent work aggregating this together. I'm curious what started you on this journey to keep this list going?
Nice concept. Similar techniques can be found in neuroevolution, where genetic algorithms optimize neural networks. One major benefit is the ability to compare the performance of both subpar and exceptional models on a relatively fair basis. Considering this evidence, I anticipate that this approach will gain greater significance as the quality of language models surpasses our existing benchmarks. However, it's worth noting that a potential pitfall arises when certain models outshine the rest of the group.
Missing WizardLM unfortunately. I was using it yesterday and found it very impressive.
Not suggesting it’s incorrect but the people who built this list also rank their own model higher than all other llama derivatives.
It's blind test, and the code to compute scores is open. So unless they are deliberately modifying the data it should be a fair ranking. Just I think openassistant llama(30B) version is better than vicuna 13b, but it is not included in the test.
Interesting benchmark!
Would mosaic mpt be useful to benchmark as well?
where is Bard?
I don't think Bard has an API available - so it can't be fairly tested.