About the 5-digit problems- I didn't make this clear but I don't think those were memorised. I think the two- and three-digit problems (all three operations) were memorised, because those are the most likely to be represented in their entirety, or close, in GPT-3's training corpus, given that they are operations that are common to very common in daily life.
I doubt that the four- and five-digit addition problems (and the single-digit, multi-op problem) were represented often enough in GPT-3's training corpus for them to be memorised. I think the low accuracy in these problems (less than 10% in the few-shot setting and near zero in the zero- and one-shot) is low enough that it doesn't require an explanation other than a mix of luck and overfitting that is common enough in machine learning algorithms that it's no surprise. e.g. we evaluate classifiers using diverse metrics, not just accuracy, because this is so common.
It is this observation, that GPT-3 did well in problems that are likely to be well reprsented in its training corpus and badly in ones that aren't, that convinces me that no more complicated explanation is needed than memorisation.
Something else. Like I say above, we evaluate classifiers not only by accuracy (the rate of correct answers), because accuracy can be misleading. e.g. a classifier can have 100% accuracy with 0% false positives and 100% false negatives. The GPT-3 authors only tested the ability of their model to give answers to problems stated as "x + y = ". They didn't test, e.g. what happens if they prompt it with "10 + 20 = 40, 38 + 25 = ". Testing for aberrant answers following from such confusing prompts has often showed that language models that appear to be answering questions correctly because of a deep understanding of language are in truth overfitting to surface statistical regularities. See for example [1,2] and many other references in [3].
Indeed, I could be wrong about rote memorisation and GPT-3 can still not be learning to perform arithmetic computations, given the tendency of language models to learn spurious correlations. There is an article about a mathemagician on the front page today, that shows how she found roots of huge numbers by finding shortcuts around expensive calculations. For instance, all sums between numbers ending in 5 end in 0, etc. I wouldn't find it magickal if a language model was finding such heuristics and that this is the "something else" that is said to be going on. However that would not be "generalisation" and it would not be learning to perform arithmetic.
In the end, I don't understand how a model can be said to know how to add two-digit numbers perfectly but not five-digit numbers. If it's performing an incomplete computation in the latter case, then what kind of incomplete computation is it performing? If it "gets it wrong after three digits" then why does it get three digits right? What's the big difference between three- and four-digit numbers that causes performance to fall off a cliff - other than the chance of finding such numbers in a natural language corpus?
As to magick- I'm writing off not the results, but the hand-waving presented in place of an explanation as magick. GPT-3 is a technological artifact designed to do one job, now reported to be doing another. This requires a thorough explanation but instead we got magickal thinking: the authors wish that their models could learn arithmetic, so they took its behaviour as proof that it learned arithmetic.
___________________
[1] Probing Neural Network Comprehension of Natural Language Arguments
https://www.aclweb.org/anthology/P19-1459/
[2] Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
https://www.aclweb.org/anthology/P19-1334/
[3] https://www.technologyreview.com/2020/07/31/1005876/natural-... (try F9 if you 're over limit)