Not an expert, but maybe using RAG/embeddings on the on-disk wikipedia would be better than finetuning on wikipedia?

Most decent LLMs probably were already trained on wikipedia, that doesn't stop them from hallucinating when asked questions about it.