I think it's appropriate to be extremely critical. The paper is basically useless. The thing that they actually measured is "can GPT-4, when given a 'question' with lots of additional information and many tries with small permutations to produce an 'answer', at some point produce an 'answer' that GPT-4 will then claim is a 5 out of 5 answer, on a dataset of extremely messy 'questions' and 'answers' from MIT coursework."

That's not an interesting thing to measure. The paper talks about it in terms that make it sound like it's a close proxy for whether GPT-4 "knows" how to do things in MIT coursework, by writing misleadingly about "fulfilling the graduation requirements" and having a "perfect solve rate." But in fact it's totally different. The result is that a bunch of people hear about this paper and get fooled into thinking that there is new interesting evidence about GPT-4's capabilities, unless they manage to read closely enough to see what actually happened.

It's not a matter of whether the results would get weaker if repeated, it's a matter of the results being totally disconnected from any useful real-world information about what GPT-4 can do, or how it can do it.

> peer review...Their null is GPT4 can ace MIT and they haven't provided any evidence to reject.

I haven't been in research for a while, but I don't think that's how peer review works. You don't always have to assume the paper's claims (especially ones as novel as this one's) as a null hypothesis, and providing a compelling refutation.

The detection of sloppy question framing, and answer feeding via the few-shot learning examples, and the problems with checking with GPT-4 itself reasonably show that there are serious flaws in multiple parts of the experiments described in the paper.

> The real conclusion is "we think the paper's results might be weaker if repeated" and that's a good result on its own.

No, they don't know enough to say that. The paper's results might be better with better experimentation! Or they might be totally false.

The conclusion they provided is accurate precisely because it focuses on the methods and not the conclusions, like this:

> One particularly worrying trend is the technique of evaluating a model’s accuracy using a language-based model like GPT-4. While a useful tool, its conclusions should never be overstated or treated as ground truth.

...and this:

> Additionally, it is extremely important to reevaluate every data point and perform basic sanity checks before using data at all, whether for training, inference, benchmarking, or something else. Given the small size of the dataset in question, a simple manual validation would have been easily within the scope of the work.

You're right that our post doesn't quite show that GPT4 cannot perform well on MIT curriculum. We try to be up front about this in the conclusion: > Our critiques are largely of the methodology and rigor of this study, not about its content. We make no claim about the ability of large language models to actually solve MIT curricula, only that this paper fails to prove it in a scientifically rigorous way. Though, as MIT undergraduates ourselves, we can at the very least say that the test set that we accessed does not, at least in our experience, accurately represent the breadth and depth of understanding required to complete an EECS degree at MIT.

>Their null is GPT4 can ace MIT and they haven't provided any evidence to reject.

If I said "2 + 2 = 4", and then gave a flawed proof, it is not your job as a debunker to prove that 2+2 != 4.

Their null hypothesis is that this paper provides sufficient and proper evidence to prove the original claim. And to that end, it seems like they've done a good job at pointing out enough serious deficiencies that - regardless of the original hypothesis's truth - the original paper cannot be used to prove it to be true.

> Our critiques are largely of the methodology and rigor of this study, not about its content. We make no claim about the ability of large language models to actually solve MIT curricula, only that this paper fails to prove it in a scientifically rigorous way. Though, as MIT undergraduates ourselves, we can at the very least say that the test set that we accessed does not, at least in our experience, accurately represent the breadth and depth of understanding required to complete an EECS degree at MIT.

Directly from TFA. Though, this does make the title somewhat click-baity. But they definitely don't claim the opposite.

>I think this is a good peer review with a too defensive tone.

Not surprising when you have undergrads calling a seemingly reputable paper almost fraud. I certainly can not blame them for the tone.

It's a criticism of methodology. It doesn't need a null hypothesis.

But, you know, I'll try that as a rebuttal the next time a reviewer rejects my papers. "Reviewer #2 has not rejected the null hypothesis" :P

The original paper claims GPT correctly answered this question. Thus the original paper is obviously wrong.

> Which invocations run in parallel? (Assuming there are enough cores.)

No context is ommitted, that's the entire question.

No, it turns out GPT-4 can not answer impossible questions. Maybe it's you that is too defensive and wants this to be false?