One problem is that the current LLMs generate code without actually running and testing it. "Here is Python syntax I dreamed up using language manipulation, but never actually ran (since I have no way of doing that without your help); it should work for you!".

If the LLM is trained on arithmetic, its arithmetic results could be more accurate than its untested code, due to that being a simpler task with simple rules over a small number of symbols with a fixed meaning.

Situations in which the LLM produces code requires the operator to be a programmer, in order to understand, and test the code, possibly fixing it, and give feedback about it. Someone who just wanted the AI to check some arithmetic doesn't want to be checking the AI's buggy code and may not even know how.