Yeah, right.
Yeah, right.
A better training regimen and better architecture optimizations have allowed smaller models to push above their weight. The leaderboard has many open 7B and 13B models that are comparable with 72B models: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
I follow your posts and comments here so I'm surprised you say that. The leaderboard at this point is pretty pointless. Lots of ways to "cheat" and get higher ranking there.
I do agree that smaller models have made significant progress, but somethings you can't just solve without adding #parameters and FLOPs. Not to mention, ctx_window is an important factor in code quality, but most OSS models (including llama 2) have pretty limited ctx, despite methods like grp and yarn.
If you look at the Chatbot Arena leaderboards there are still decently-high ELOs for 7B models.
Except for the leaderboard. Its all but useless, not just because of the data contamination/cheating but because the benchmarks themselves are flawed. They are full of ambiguity/errors, and they dont even use instruct formatting.
Phind at times is hampered by whatever it is they're doing in addition (RAG?). It is still phenomenal, though. I regularly find myself using Phind to grok assembly code or learn Typescript.
I pay for it and for chatGPT and I find copilot much worse.
For code review, I tend to engage Copilot Chat which probably uses GPT4 more often? https://github.com/orgs/community/discussions/58059#discussi...
It's also true that specialist models still need to be sufficiently large to be able to reason well, but we've observed diminishing returns as models get larger.