Without having read the article, I would say it demonstrates that benchmarks alone, without knowledge of how a model was trained (c.q. constructed), cannot reliably determine whether or not a model has certain capabilities.
10 karma · joined May 15, 2024
Although I've been a daily Emacs user for +25 years, I've only occasionally invested time in tweaking ICR (messing with the basic editor stuff you use hundreds of times a day can really get in the way of getting things done), and the sheer number of packages and sub-packages in the ICR space meant I only had a hazy idea of how they all related and (are they complementary, alternatives, successors?). Your VOMPECCC blogpost does a terrific job clarifying that!
Remind yourself often of the achievements you are proud of.