Comment by vessenes | Hacker News Reader

Comment by vessenes | Hacker News Reader

Parent Full thread

I don't think they're meaningless; they have a few benefits:

1) He doesn't have an ax to grind / an LLM to pimp out, so he's relatively even-handed

2) He uses the same (secret) test data for each model, so his testing is resistant to cherry-picking/finetuning on tests

3) He likes weirdo role-play prompting, so he has a very good sense of the edges of refusal and alignment tuning

4) He picks up stuff well before it hits the only other fair testing I know of, the chat arena

5) I think asking stuff in German is at worst neutral, and at best useful for testing capacity in edge cases.

Practically speaking, his 'preferred' non-giant models, Nous-Capybara-34B and Mixtral both are excellent in comparison with some of the others he looks at, and good recommendations.

That said, I'd like to see a test suite that GPT-4 fails at, or struggles at, at least. And, it would save him a lot of time if he could get something automated together, it's clearly a lot of effort to hand test all those models.

I agree with all except using german, because some model might be better at german but it doesn't necessarily means they are better overall.

For example I'm pretty sure Mistral models are better at french, so doing a benchmark using french only would be advantageous for them.

If you want to compare all models, better use english. Because now his benchmark just show which models is better at german.

That being said, it's still a very welcomed benchmark.

Any tests that are unfalsifiable and can’t be reproduced are meaningless when it comes to gauging the performance of LLMs (and most other things). I could also post on a subreddit and say I have a secret set of German tests that may or may not exist and and I like these models, but that does nothing to advance the science of evaluating these things. If you want to evaluate human preferences, you can use chatbot arena, which can be gamed, but at least reflects more than what one guy says to be true. And this is with me agreeing that Nous-Capybara is also a good model. But don’t take my word for it because it’s not worth very much!

I think we agree on almost everything about this - which is to say, probably both you and I think the Wolfram Ravenwolf tests are just about useful enough to indicate whether or not a model might be worth downloading, but certainly not enough to justify spending money, say, or planning around. So, yes, I'm with you.

I agree that better ways to evaluate models would be super super useful, and benchmarks like MMLU and whatever's next will continue to be helpful ("real" science). And, it seems like there may even be some benefits for models to training to 'ace the test' more broadly, which is interesting, and ties to some educational theories in teaching humans.

However, one area that open tests can't excel in is in this "fair" evaluation arena -- and I do think that private tests have some value there, to the extent that they can show utility and maintain trust. I don't make any claims about German sex role-play being a good or bad start for these, though.

I think there’s room for private tests, but those should probably be done by some kind of independent standards body. LLMs are an incredible advancement of machine learning research, but we diminish that when we let these kind of wholly unscientific evaluations predominate and especially in open source, where so much great work is otherwise being done to combat the monopoly tactics of big tech.

Reply on news.ycombinator.com