I've watched most of Karpathy's videos but it's been a while so if I ask a dumb question it's because I'm at a very basic level from a while ago.
To try to turn this into something concrete, let's say I want to store as facts the 50 state capitals. We'll do the most basic tokenization possible, so I have ['AL': 1, 'AK': 2, 'AZ': 3, ... 'WI': 49, 'WY': 50, 'Montgomery': 51, 'Juneau': 52, ... 'Cheyenne': 100]
(Do I need an embedding matrix here? I feel like I need a 100x64 embedding matrix, one for each "token", even for those very first linear examples? Maybe I need two 50x64 matrices?)
But then I'm learning randomly initialized V and K matrices by going through and dotting my 'AL' row by K and my 'Montgomery' row by V and then dotting them together, and going through all of the rest of the states similarly and then optimizing a loss function until I get a V and K that gives me good performance?
I kind of wanted to go one step further and adding two more tokens, a 'StateHasCapital' token and a 'IsCapitalOf' token (say 101 and 102), and somehow use that to learn with compound keys ['StateHasCapital'+'AL': 'Montgomery'] - if I'm really learning facts, can I train on all 50 States->Capitals and maybe 25 Capitals->States and then as a test to see if I'm learning "facts", use the remaining capitals that I didn't train on to see if I can look up what state has the capital city of Olympia?
I might be conflating a whole bunch of steps that don't make sense in that initial linear example and adding complexity, and maybe trying to fit too much into the later examples too (I don't see the 'Query' matrix in the any of the parts of the post that get into self-attention, and maybe this post is just about "here's how memory works, not how LLMs work"