For example, they manually rewrote the question:
> Find the derivative of the function using the definition of a derivative. f(x) = (x2 − 1)/(2x − 3)
To
> Use Sympy to find the derivative of f(x)=(x*2-1)/(2*x-3)
Which completely changes the question, and also their rewrite ensures that the AI doesn't fail to parse any data etc.
> Calculate the probability of getting a two-pair poker hand.
Was rewritten to, this, close to code translated to human text:
> A hand is a set of 5 cards that are drawn randomly from a standard 52 card deck with 13 ranks of 4 cards each. A two-pair poker hand is a hand that contains 3 unique ranks, where no more than 2 cards in the hand can share the same rank. That is, 3 or more cards cannot share the same rank. Write a program that generates simulations for calculating the average probability of getting a twopair poker hand.
They even changed the question to allow simulations and not just exact answers. So then the AI is "correct" here when it generates 1000 random samples of taking 5 elements from a set of 52 elements and seeing how many are two pairs.
Not sure what the point of this paper is then. It doesn't do anything novel with the mathematics as it either just uses libraries or it doesn't really solve the problem, nor does it to anything novel with text recognition.
It's just as well. If an artificial neural network was actually doing undergraduate mathematics I would have dropped out of my mathematics PhD yesterday.
Now the challenge is to mimic the human creating the "right" question with high fidelity so the whole ML system returns appropriate search results.
This was always the hard part. If you can generate the "right" question then you can generate the code, and since there are already many good math libraries for solving undergrad problems that is all you have to do.
I think the first phrasing is asking you to directly use the definition:
f'(x) = \lim_{h\to 0}{{f(x+h) - f(x)} \over h}
FWIW, this is also not what I would expect ‘University-level mathematics’ to mean. The second meaning fits into high school, at least where I grew up. I was expecting proofs to be honest.
It's not doing anything more than that, in any of the examples shown in the paper.