Everyone cites these kind of examples as LLM beating some test or other as some kind of validation. It isn’t .
To me that just tells that the tests are poor, not the LLMs are good. Designing and curating a good test is hard and expensive.
Certifying and examination bodies often use knowledge as a proxy to understanding or reasoning or any critical thinking skills.they just need to filter enough people out, there is no competitive pressure to improve quality at all. Knowledge tests do that just as well and are cheaper.
Standardization is also hard to do correctly, common core is a classic example of how that changes incentives for both teachers and students . Goodhart's law also applies.
To me it is more often than not a function of poor test measurement practices rather than any great skill shown by the LLM.
Passing the CFA or the bar exam while daunting for humans by design does not teach you anything practicing law or accounting. Managing the books of a real company is nothing like what the textbook and exams teaches you .
—-
The best accountants or lawyers etc are not making partner because of their knowledge of the law and tax. They make money same as everyone else - networking and building customer relationships. As long as the certification bodies don’t flood the market they will do well which is what the test does.