My biggest gripe with this paper is how unclear it is on its methods. How many rewrites per question? How did they select the solved questions? They state they used a model trained on language and fine-tuned on code, but fine-tuning could describe either their or codex's process.
The biggest hint in favor of fine-tuning is that AFAICT, their dataset adds up to 205 questions but they've solved 235 questions. In which case I'd suspect overfitting on question form. In intro level math, problems are usually in template form and solving them boils down to matching on the correct template and slot filling the answer.
To prove whether it's been fine-tuned, people with davinci codex access should try to see if the can replicate this.
To prove it's not overfit, authors should release their training dataset if there is one and allow people to test with it.
How many parameters does davinci codex have? The original codex was 12 biliion IIRC and certainly wasn't this capable.
---
Some of the answers look wrong or incomplete.
Table 221 seems to be missing a factorial, such leniency suggests they must not be automatically scoring this.
Not sure what's going on in Table 144. Table 44 too.
In 42 and 47, particularly 42, the solution program seems incomplete.
211 is impressive, it also writes code for LU decomposition, permutations, runge kutta, card combinatorics and other clever stuff.
Even though for most of the more challenging problems, hard part was pre-digested or worse, it was hand fed the answer's algorithm, there were a few where the network had to have had a solid understanding of the math and code requirements. A program like they claim would revolutionize and hugely simplify mathematically and algorithmically involved programming.
The biggest cause for worry that it might have overfit is that it works with no sampling at all and gets a perfect score (they claim).