Hacker News Reader

I wonder how well it would perform in accuracy if given a large number of simple but lengthy sums like 13453 + 53521. Increased set size would move it beyond simple input/output memorization. Although if it recurses properly and carries the digit it could be text parsing and have an accurate but probably very inefficently written math parser.

View on HN

> Although if it recurses properly and carries the digit it could be text parsing and have an accurate but probably very inefficently written math parser.

I suspect this is how many humans do arithmetic (especially considering how many people conflate numbers with their representation as digits). So if GPT-3 is doing that, that's pretty impressive.

You don't have to wonder. In their paper: https://arxiv.org/abs/2005.14165 they state it has 0.7% accuracy on zero shot 5 digit addition problems and 9.3% accuracy on few shot 5 digit addition problems.

By the way: Arithmetic accuracy is better if dollar sign and commas are added (financial data in the training set):

http://gptprompts.wikidot.com/logic:math

You do have to wonder, because as that section states, the BPEs may impede arithmetic, and as we've found using the API, if you use commas, the accuracy (zero and few-shot) goes way up.

Reply on news.ycombinator.com