>> According to the paper, Computer Vision research is notably more affected by the syndrome it outlines than other sectors, with the authors noting that Natural Language Processing (NLP) research is far less affected. The authors suggest that this could be because NLP communities are ‘more coherent’ and larger in size, and because NLP datasets are more accessible and easier to curate, as well as being smaller and less resource-intensive in terms of data-gathering.

I'll have to read the paper to see what exactly it says on this but my knowledge of NLP benchmark datasets is exactly the opposite: the majority are simply no use for measuring the capabilities they're supposed to be measuring. For example, natural language _understanding_ datasets are typically created as multiple-choice questionnaires, so that, a) there is already a baseline accuracy that a system can "hit" just by chance (but which is almost never noted, or compared against, in papers) and b) a system that's good at classification can beat the benchmarks black and blue without doing any "understanding". And this is, indeed, what's been going on for quite a while now with large language models that take all the trophies and are still dumb as bricks.

To make matters worse, NLP also doesn't have any good metrcis of performance. Stuff like the BLEU scores are just laughably inadequate. NLP is all bad metrics over bad benchmarks. And NLP results are much harder to just "eyball" than machine vision results (and the models are much harder to interpret than machine vision models where you can at least visualise the activations and see... something). I think NLP is much, much worse than machine vision.