Just some days ago researchers from Peking U and Microsoft published a paper[0] saying they can access "knowledge neurons" in pretrained embeddings that will enable "fact editing"[1].

[0]: https://arxiv.org/pdf/2104.08696.pdf

[1]: https://medium.com/syncedreview/microsoft-peking-u-researche...

I thought that Transformers were a type of associative memory.

Relevant paper from Misha Belkin's group https://arxiv.org/abs/1909.12362