1) He doesn't have an ax to grind / an LLM to pimp out, so he's relatively even-handed
2) He uses the same (secret) test data for each model, so his testing is resistant to cherry-picking/finetuning on tests
3) He likes weirdo role-play prompting, so he has a very good sense of the edges of refusal and alignment tuning
4) He picks up stuff well before it hits the only other fair testing I know of, the chat arena
5) I think asking stuff in German is at worst neutral, and at best useful for testing capacity in edge cases.
Practically speaking, his 'preferred' non-giant models, Nous-Capybara-34B and Mixtral both are excellent in comparison with some of the others he looks at, and good recommendations.
That said, I'd like to see a test suite that GPT-4 fails at, or struggles at, at least. And, it would save him a lot of time if he could get something automated together, it's clearly a lot of effort to hand test all those models.
I agree that better ways to evaluate models would be super super useful, and benchmarks like MMLU and whatever's next will continue to be helpful ("real" science). And, it seems like there may even be some benefits for models to training to 'ace the test' more broadly, which is interesting, and ties to some educational theories in teaching humans.
However, one area that open tests can't excel in is in this "fair" evaluation arena -- and I do think that private tests have some value there, to the extent that they can show utility and maintain trust. I don't make any claims about German sex role-play being a good or bad start for these, though.
For example I'm pretty sure Mistral models are better at french, so doing a benchmark using french only would be advantageous for them.
If you want to compare all models, better use english. Because now his benchmark just show which models is better at german.
That being said, it's still a very welcomed benchmark.