valid point. i'm not a statistician but it does seem to me to be statistically significant.

According to the paper, the validation set has 107 problems so it's a difference of 6 problems solved, not just 1-2.

Also, it's not a typical eval with multiple-choice answers — these are difficult coding problems from competitions where the answers are strings or integers, so the odds of "guessing" a correct answer are very low.