The former- it's something of a party trick for him. Most people can chunk up to 3 digit numbers, so you really only need to remember 4 numbers (original two, result, and temp). When I was little people would try to beat him on a calculator, and rarely won.

His talent was mainly that he could factor super quickly, and had the times tables memorized up to some absurd level. So that just cut a lot of thinking out of the task.