Retries, answer verification systems, formatting and parsing. Reasoning and validation of reasoning with natural language is a key challenge.
Closed feedback loops to improve the quality of reasoning for LLM's requires human-in-the-loop tools.
Prompting methods are often a hit and miss as the same prompts can lead to variable quality outputs.