If I create 1000 predictive models, with random parameter settings, some of those models will outperform others. In fact, the accuracy rankings of the models given a single hold-out validation set will form their own distribution. It is difficult, if not impossible, to tell if one model outranks another b/c it is actually better or just occupies a different place in that distribution due to the random properties of the hold-out set.

So, why do you believe this isn't happening when hundreds of people submit their models to be scored against a secret hold-out set? Is there any evidence that the hundreds of entries are forming a better distribution of accuracy scores than random models? Is there evidence the winners of these competitions truly have better models and don't just have a luckier placement in a random distribution of accuracy scores?