Great idea. I thought about similar things like create a traditional NLP chatbot and attach an whatever LLM to the answers.