Before getting too hung up on memory and seeing these as key value stores, or self-attention is suppression of interference, recognize that a lot of experts and people in these areas disagree with this analogy for fairly deep reasons (I link other resources and discussion here: https://news.ycombinator.com/item?id=50131741). It is still a useful analogy, but it is incomplete and weak in many ways when it comes to the actual details of training.

Also make sure to understand how self-attention layers are actually applied and the exact math and matrices involved. E.g. in PyTorch you are going to be doing sa(x, x, x), where x is the previous layer's output (essentially, embedded tokenization), and sa() does quite a few matrix multiplications. In cross-attention, you are doing sa(x, x, y), where y comes from a different part of the architecture.

Basically, read broadly here and don't get stuck on a single analogy / intuition for what these layers are doing. Focus on learning the actual math of the layer and the network first, and then look for analogies that help pump your intuition. But don't try too hard to force a single analogy onto the math.

EDIT: Also modern AI can provide useful insight into posts like this, I highly suggest using them when learning these things. E.g. https://chatgpt.com/share/6ad3aa3d-d110-83ea-a8c5-8653b34743...