You don't have to wonder. In their paper: https://arxiv.org/abs/2005.14165
they state it has 0.7% accuracy on zero shot 5 digit addition problems and 9.3% accuracy on few shot 5 digit addition problems.
You do have to wonder, because as that section states, the BPEs may impede arithmetic, and as we've found using the API, if you use commas, the accuracy (zero and few-shot) goes way up.