In my experience the biggest roadblock to continuous conversation is context length. It fills up and the LLM starts forgetting parts of the conversation. Most tools don't even tell you that this has happened. But if you keep that in mind, that there is a buffer there filling up, you can massively improve the quality of the output.