Have you thought about using llm-wiki? It's pretty powerful and works well.
I’ve seen some good research where xx% of the time, all LLMs essentially just get lazy and won’t actually look things up.
Been working on an informational retrieval system similar, but endless Nerfs combined with non deterministic and finicky behaviors make it hard to reliably depend on.
How’s your luck been with LLm-wiki?