As humans we know when and how to interface with a better adding device, such as a calculator. Could a LLM not do the same?
There's no way for it to separate calculations from other transformations and thus it cannot delegate calculations to a different subsystem.
This can also be seen as a security feature, as arbitrary calculation is by nature unbounded in terms of complexity and memory use. There are calculations that seem simple, never exceed a reasonable value range, yet take ages to compute. Since it's impossible to identify such functions by simply looking at them, it'd be a great way of basically performing a DoS-attack on the model.
Correct that it's impossible because you would need to solve halting problem to do it but you could set energy/time limits for query and just stop it when you reach that limit.
What you say is achievable only if another system external to the model takes some tagged model output, makes computations or lookups, and feeds the results back to the model in the form of text input.
Then it's game on for the model to trigger some form of code execution through this external system and escape the jail...