A lot of this can be solved by using clever prompting and allowing the LLM to use external tools - see the ReAct paper https://arxiv.org/abs/2210.03629

I believe OpenAI is going to introduce this to ChatGPT soon as they launch the plugin system.