Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by smt88 | Hacker News Reader

Parent Full thread

smt88·

Is a 6-point difference substantial in a statistical sense? To me it seems small enough to be noise.

View on HN

valid point. i'm not a statistician but it does seem to me to be statistically significant.

According to the paper, the validation set has 107 problems so it's a difference of 6 problems solved, not just 1-2.

Also, it's not a typical eval with multiple-choice answers — these are difficult coding problems from competitions where the answers are strings or integers, so the odds of "guessing" a correct answer are very low.

I couldn't find the exact test methodology but it would be trivial to rerun it multiple times to get a good average (as well as all other inputs you need to test statistical significance)

Depends on how many questions there is.

To me it sounds huge, but if there are only 16 questions, then yeah, it’s not significant.

Especially if you consider that the more one writes online about the CodeContests benchmark, the more training data including the CodeContests benchmark data is fed to the next iteration of GPT. We could be doing massive scale over-fitting without realizing it.

I haven't seen any evidence of over-fitting in the sense of the model getting worse on new examples. Conversely, it seems that we're gradually fitting the models on more and more types of coding questions, such that it's gradually becoming more and more difficult to find well-defined problems that humans can solve easily but AIs cannot.

Right, I would think a 20-30% difference would be significant.

That's not what significance means in a statistical sense

Reply on news.ycombinator.com