Differential Equations with Transformers: Deep Learning for Symbolic Math
medium.com
medium.com
Arxiv paper: https://arxiv.org/pdf/1912.01412.pdf
The performance metric is the number of equations/integrals correctly solved over a held out test set, generated randomly. This is done using an external tool (SymPy), which some might consider "cheating" (but you need a way to test, don't you?). Anyway, keep in mind that checking a solution for correctness is much easier than finding one.
Another issue was about using timeout for Wolfram. This is now discussed in the appendix (it makes very little difference).
During review, an interesting point was discussed: out of distribution generalization, or how performance depends on the random problem generator we used. This is now measured and discussed at the end of the appendix.
Your appendix E poses a really interesting question that I hope future research addresses. The distribution of integrals that appear on math tests are likely to closely match FWD, BWD and IBP because that's how professors write problems for students. However the distribution of integrals that arise in physics and engineering is quite different and includes a lot of ones that aren't solvable in closed form. I wonder what the performance of the system would be on the "engineering distribution." I also wonder what you would have to do to find out what the engineering distribution was...
For integrals, solvability depends on the function set you work with. Since we use elementary functions on the real domain, a lot of integrals have no solution. We could have gone for a larger set (adding erf, the Fresnels, up to Liouvillian functions). This would mean more solvable cases.
As for the engineering distribution, no one knows what it is. The best we can do is to generate a broader training set, knowing that it will generalize better (this is the key takeaway of our appendix). BWD+IBP is a step in this direction, but to progress further, we need a better understanding of the problem space, and issues related to simplification. We are working on this now.
Here is something to think about if you want more solvable cases: even in the case of sine and cosine, solving a differential equation really means reducing it to a combination of the solutions to simpler differential equations. The sine function can be defined as the solution to a particular differential equation, as can all of the fancier functions. So in a sense it's kind of like factorization, where you have "prime" equations whose only solution is a transcendental function defined as being their solution, and "composite" equations whose solutions can be written as a combination of solutions to "prime" equations. So really all of the rare functions belong to the same general scheme.
At test time, you usually want a test set from the same distribution as the training data (or at least related to it in some controllable way), or it becomes very difficult to interpret the results.
Suppose my test set come from a different and unknown distribution (real problems sampled in some way).
If I get good results, is it because the training worked, or because the test set was "comparatively easy"? If I get bad results, is it because the model did not learn, or because the test set was too far away from the training examples?
So the paper does "guess and check," and only returns answers that pass the check, i.e. are exactly right.
There are some problems where all the checks fail, i.e. the paper can't generate an answer. And there are also questions where Matlab / Maple can't find an answer either.
* It's a little more complicated; e.g. if f = cos^2, and the derivative of F is 1 - sin^2, I need to recognize they're the same. But it's safe, in the sense that if I can't determine whether they're the same, I can reject the solution and move on to the next one.
Integration is a completely different ball game. Given an arbitrary expression, most likely it's not symbolically integrable, i.e. it's integral can't be expressed as a combination of known functions. That's why there are lots of functions defined as "the integral of blah."
Even given an expression that's known to be symbolically integrable, it's quite the puzzle to figure out what the integral is. There's no straight forward algorithm, as with differentiation. It boils down to "try integration by parts; try this; try that; maybe do some clever algebraic manipulation to get it into a form where you can try one of those things; look it up in a book of integrals to see if it's there, or something similar is there; if all else fails, guess and check."
If this new system can integrate some equations that other methods cannot, I welcome it. It is almost instantaneous to check the solutions, so what is the problem?
I imagine the devs at Wolfram Mathematica are already working into adding this new kind of solvers into their overall integrator.
Then for more complex math, can't it be argued that this is just recombination of memorized theories and algos, but using some kind of random search?
I'm not sure math involves as much symbolic reasoning as we like to think, but I could be wrong. The same goes for programming - mostly just regurgitating known algos (which break into smaller memorized sub-algos). When presented with something novel, generally just use random approaches until one fits. A few people in history who had to come up with "the first" such programs had some novelty, but even those were mostly just applications of known / memorized solutions in a random way (e.g. quicksort).
so no it's not this straightforward except in basically toy examples.
http://www-sop.inria.fr/cafe/Manuel.Bronstein/pmint/index.ht...
Clearly, if your generator for equations is limited the system can learn the dictionary lhs (integral) => rhs (derivative) to go rhs => lhs. (integrate)