Yes, we can calculate the distribution and find the likelihood that a given team would beat another if the data is resampled. Here's a detailed example: http://www.tiberius.biz/pakdd07.html

The size of improvements seen in competitions vs previous best performing models (eg Allstate competition showed improvement of 270%) is well beyond that explicable by chance. Also: the test set size in most competitions is hundreds of thousands of rows - sampling error is very small in these cases.