at present, my understanding is that it is purely linguistic. If you ask "what do you call the wife of a king", it will answer "queen" because that is the highest likelyhood next token. If you ask "what is 2 + 3", it will answer "5" because that is the highest likelyhood next token purely from a linguistic point of view, i.e. no math has been performed.

All the emergent properties of chatgpt being so good, is that it is very good at predicting word sequences that, when evaluated by a human, sound like it is doing all sorts of higher level logic.