Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by sbierwagen | Hacker News Reader

Full thread

sbierwagen·

Doesn't do so great on the leaderboards: https://tatsu-lab.github.io/alpaca_eval/

View on HN

Depends on the benchmark. Does well on other metrics when compared to open models.

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

Given that HellaSwag performance seems to correlate with reasoning ability more than other benchmarks, Falcon certainly look promising! Hopefully this is a clean result and not the product of dataset contamination.

I've given it a try, to having a chat is good, to follow langchain prompts it's not.

I guess it depends on the type of work you want to extract from it.

That leaderboard is incorrect

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

Actually it's HF's leaderboard that's bugged. Falcon is only on top since their MMLU scores are bugged across all LLaMA models: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

I really like the fact that the leaderboards are almost identical when using claude or GPT-4 as evaluators.

If a less powerful model can be a good decider of the better answer between two more powerful models, it opens up a lot of research opportunities into perhaps using these evaluations as part of an automated reinforcement learning process.

Its a pretty terrible model, I wouldn't use this for anything at all. Vicuna 1.1 outperforms it in all my tests.

What tests are you running? Anything outside of the alpaca eval test suite?

That leaderboard does not line up with my personal experience with those models at all...

Reply on news.ycombinator.com