Yes - for example adding memory to the attention mechanism (similar to RETRO or Memorizing Transformers paper)