It looks at the pattern of a bunch of unique tokens in a dataset (in this case words online) and riffs on those patterns to make outputs.
It will never learn math this way, no matter how much training you give it.
BUT we have already solved computers doing math with regular rules based algorithms. The way to solve the math problem is to filter inputs and send some to the GPT NN and some to a regular algorithm (this is what google search does now for example).
GPT is an amazing tool that can do a bunch of amazing stuff, but it will never do everything (the metaphor I always give is that your pre-frontal cortex is the most complex part of your brain, but it will never learn how to beat your heart).