I have my local chatbot set up to automatically run shell commands that start with "$." It wouldn't be too hard to check the status and have the LLM try to debug it if something goes wrong.

This to me seems much more flexible and likely to work than training a neural network to do arithmetic.