I've considered this as well, as many others have. I tend to think that A: this is moreso a feature rather than a bug, and B: this is almost impossible to measure- it's a boogeyman imo.

In other words, combating this is impossible if it's happening. Measuring whether its happening is also not plausible, for small fish running bespoke benchmarks suites. I think we (consumers) have hit a pretty clear stride of; new model releases > some subset of evals/benchmarks are saturated > new, harder, more niche evals and benchmarks take their place. That doesnt seem unhealthy to me.