They can explain addition to you but they can't perform it without calling out to an external API.
In dealing with complicated code, this is a very insidious problem, because they can often write functions that compile and run, but which miss edge cases because there is no actual logic behind them.
[edit] I should add that a sufficiently large NN may be able to create an entire computer inside itself that can run code, but even that would not be reliable because it will have arrived at the gating by inference rather than starting from a deterministic root.
Best evidence for this is the construction of natural numbers in traditional math* with `successor` function (A. There is a one (1). B. There exists a successor of one (succ(1)) called 2.) — everything else follows from that using formal logic, and that's why everything else is generally out of reach for generative ML models (eg. summation can be defined in multiple ways, one of them being successive application of a successor function the right number of times on one of the operands).
* Modern approach is to use set cardinality instead, but the essence is the same: something has to be memorized (axioms), and then logic is applied.
Take a 3-year-old who's holding two pieces of candy. Say, "I'm going to take these and give you two pieces back." Then give him one piece back. He's going to start crying. Perhaps he can't explain why, exactly. But if you ask him why it's not fair, he's going to tell you that two is more than one. Ask why is two more than one: It doesn't require succession. It's a fact of the reality we inhabit. We can measure it with our hands, our eyes, its weight. We can hear two sounds, two voices at once and differentiate them (non-blocking, nonlinear processing). AIs don't inhabit any reality grounded in the physical constraints they purport to understand, therefore they can never arrive at the conclusion that 2 > 1. They just appear to "know" it because it's part of the data set.
But once you understand that 2 > 1, then 3 must be > 2. If acquisition is hardwired into our brains, then so is the capacity for logically explaining why Y > X.
Food for thought- would a savant use the same process? Or have they somehow recruited more of their brain to where they can memorize much larger problems.
I think savants usually solve problems in the most deductive way, using reasoning that leads to other reasoning. I went to an elementary school in the 80s where more than half the kids would now be labeled autistic... some got into math programs at colleges by the age of 12. I believe it's all pure reasoning, not like some magical calculator that spat out answers they didn't understand the reasons for.
[edit] If you meant: Do savants solve problems by recursively breaking problems into smaller and smaller problems, then yes. But the breaking-apart-of-problems is actually the hard problem, not the solving.