Artificial Analysis is paid by AI labs and they don't publish the full method. To me it's meaningless, let's wait for other benchmarks, but if they're correct the intelligence increment is negligible.
https://artificialanalysis.ai/methodology
If they published their private benchmarks the Chinese labs would simply train on the data to make it look like they are close to the frontier and anyone who uses them regularly knows they are far, far behind and always have been.
It's not perfect. But it's the best we have. You are welcome to provide a better source.
First, the way you phrased your argument makes it sound like it benchmaxxing is mostl a problem from the chinese models. As recent articles seem to have shown, it's smaller models, closed and open weights, which are benchmaxxed. That has nothing to do with the country they're coming from.
Second, I feel like your argument about chinese models being far behind is not quite relevant to the argument and was mostly added as a jab against these models.
Third, arguing against publishing the benchmarks is not the same as arguing FOR using private benchmarks. You don't have to use benchmarks at all, especially when models are so hard to quantify anyway. Simply shrugging and saying "well, no way around that" is not a solution. In addition, there are other, better ways to prevent benchmaxxing than simply "not publishing your solution". The recent maxxbench benchmark has shown one example.
Fourth, you don't need to provide a better alternative to criticize something. Criticism is valid even if you don't have a better alternative yourself
And lastly, I actually feel like there IS a better alternative than AA - not using aggregate score at all. If you need a model for coding, look at the individual benchmarks themselves for a / very / rough overview of how a models is doing. Trying to boil down so many different areas and capabilities into a single score, which is then weighted by little more than subjective vibes from the people behind AA is bound to be too little information and too much information loss. This, I personally feel, seems to mostly exist as a meaningless leaderboard for marketing purposes.