Deepmind has been doing some interesting work around Retrieval Enhanced Transformers (RETRO) models [1] that might be relevant in this context.

[1] https://www.deepmind.com/blog/improving-language-models-by-r...