You could have invented transformers from memory
alexwang.ai
alexwang.ai
The concepts of fidelity/recall and the images/animation at the top reminded me of this small playful project that touches upon these topics: https://nicobrenner.github.io/lossy-git/
What is d_k - I assume it's the dimension of the k vector. Similarly, I assume d_v is the dimension of the v vector. What is the "i" suffix for here at this early stage? It doesn't need to be there, it's just confusing.
You say the keys are "used for lookup based on similarity". What does this mean? Similarity to what? The existing keys? Something else?
Your first example then introduces the identifier "M", which seems to be the number of memories. Again, this is not defined. It speaks again of similarities, and introduces "weights". But the matrix Mxd_k is a matrix made of all your keys, which aren't "weights" - or at least this term is new. The term "tokens" makes an appearance around this time also - something else that needs defining.
I think you need to go through the text of this article, with an eye to someone who is reading it for the first time. Define every term you use, remove unnecessary notation - and also perhaps explain at the start why one would be interested in a "memory store" of this nature.
There is a place for something that really explains what's going on here. I've yet to find it.
and others. Hippocampal sparse coding/orthoganalization/pattern separation and completion/etc may be interesting search terms for you, if you are unfamiliar. If you do happen to be a hippocampal nut like I used to be during grad school, I salute you
Curious to hear more about the stack for your blog? Thanks
Please write a proper Blog post.
The Twitter thread is a summary of the blog post.
You can just read the blog post.
I've watched most of Karpathy's videos but it's been a while so if I ask a dumb question it's because I'm at a very basic level from a while ago.
To try to turn this into something concrete, let's say I want to store as facts the 50 state capitals. We'll do the most basic tokenization possible, so I have ['AL': 1, 'AK': 2, 'AZ': 3, ... 'WI': 49, 'WY': 50, 'Montgomery': 51, 'Juneau': 52, ... 'Cheyenne': 100]
(Do I need an embedding matrix here? I feel like I need a 100x64 embedding matrix, one for each "token", even for those very first linear examples? Maybe I need two 50x64 matrices?)
But then I'm learning randomly initialized V and K matrices by going through and dotting my 'AL' row by K and my 'Montgomery' row by V and then dotting them together, and going through all of the rest of the states similarly and then optimizing a loss function until I get a V and K that gives me good performance?
I kind of wanted to go one step further and adding two more tokens, a 'StateHasCapital' token and a 'IsCapitalOf' token (say 101 and 102), and somehow use that to learn with compound keys ['StateHasCapital'+'AL': 'Montgomery'] - if I'm really learning facts, can I train on all 50 States->Capitals and maybe 25 Capitals->States and then as a test to see if I'm learning "facts", use the remaining capitals that I didn't train on to see if I can look up what state has the capital city of Olympia?
I might be conflating a whole bunch of steps that don't make sense in that initial linear example and adding complexity, and maybe trying to fit too much into the later examples too (I don't see the 'Query' matrix in the any of the parts of the post that get into self-attention, and maybe this post is just about "here's how memory works, not how LLMs work"
I didn't really follow the article to its conclusion. Surely the logical conclusion of "higher powers are better" is to take the limit as power goes to infinity - which is equivalent to just finding the most similar key and then returning the corresponding value. Why would you take a combination of all powers, when you know that higher powers work better?
Nothing in the article explained why we need "a memory" to work in this particular way that leads to it becoming a kind of transformer.
Talking about context tokens storing memories makes no sense. By definition, they are not memories, because they are all from the current prompt!
The whole blog post feels like a just-so story designed to make you think transformers were obvious.
The article has the answer. The problem with that is that it is no longer differentiable.
The limit to infinite powers is to use the exponential function, which has weighted powers up to infinity. But you still need a kind of smoothing factor so that you don't end up just picking the closest one and losing differentiability, which they call beta and is picked based on how many memories you've stored.
Where it gets tricky is that the reader might not be part of the audience intended by the writer.
It does mean "you, the reader" or "you, any random reader from my intended audience" (e.g. fellow PhDs) in articles like these.
Using it to mean "one" is a different use, in more formal talk.
(Even the audience for, say, 3b1b's explanation of what a transformer is, is niche.)
Recently, Jensen Huang reportedly said he doesn't know where he lives (the address) with a straight face. This 'you' situation here is kind of like the Jensen situation: neither party understands what it's like to be the other party.
Oh you know there is gonna be maths in this one. (clicks link)
"Here are the terms we will use throughout: a 'memory store' is a function f..."
Yes. Ha ha! Yes!
I recall awhile back hearing someone ask whether intelligence was actually the same or similar to “compression” or storing all of this knowledge in weights - IE the ability to create something from reason or weights or relationships / connections. This article makes it even more interesting!
(For anyone not familiar with 3B1B, it's a YouTube channel that visualizes intriguing math concepts to explain them intuitively!)
I got to this part:
> If the keys are random unit vectors
And hesitated here. It seems then the article is predicated on us trying to store "white noise". However in the real world, data has much more structure than that. Every bit of structure is an opportunity to compress more. So all the conclusions here would be an upper/lower bound rather than a true estimate.
I assume this is accounted for by:
> In fact we haven’t actually considered the training data at all, which affects what actually is being represented in the keys and values
but it understates the point: much of the compressability as well as intelligence in the models come explicitly from the structure in the data. So it's very interesting you can derive MLP and transformer architecture even absent that. But the real leverage comes from the interaction of those things WITH structure in the data.
> Glanville, R. (1976). What is memory, that it can remember what it is? In R. Trappl et al. (Eds.), Recent Progress in Cybernetics & Systems Research, Volume 7 (Proceedings 3rd European Meeting on Cybernetics and Systems Research). Washington, DC: Hemisphere Press.
INSIDE EVERY WHITE BOX THERE ARE TWO BLACK BOXES TRYING TO GET OUT’ by Ranulph Glanville Portsmouth Polytechnic, England
The nature of the blackness of a black box (when an observer does not know what is going on inside the black box) and its whiteness (when an observer believes that he knows what is going on inside the black box) is considered and is applied to a description that is built by an observer of that black box. It is shown that while whiteness may be believed to have been reached within a system, such a system, nevertheless, remains black on the outside. This understanding is considered because it sheds light not only on black boxes and knowledge, but also on certain very basic cybernetic and broad psychological concepts.
KEYWORDS:conceptual system, individual, behavior, black box, white box, cybernetics, feedback, input, output, Object, self-reproduction.
the reference is:
Glanville, R. What is memory that it can remember what it is? In R. Trappl, (Ed.), Progress in and systems research (vol. 4). Washington, D.C.: Hemisphere, 1978.(c)
**
Also here: https://constructivist.info/radical/papers/glanville/glanvil...
[7] R. Glanville "What is Memory, that it can remember wheat it is?" in R. Trappl (ed.) "Progress in Cybernetics and Systems Research Vol 4", Washington DC, Hemisphere, 1978
... very odd ...
For an LLM, the keys are the tokens in your context, and the values are … what exactly?
Q = XW_q
K = XW_k
V = XW_v
From there, you can think of K and V like keys and values in a Python dictionary, and Q as the key you pass (e.g. d[Q] in Python to return some value V). The difference in an attention layer is that the query is matched to each key using a dot product, and rather than getting a single value back you get a weighted sum of values.
I understand you meant to compare it to dicts, but this is more like (the only?) similarity to outer product memory. Worth calling out since GP is confused about this connection specifically.
Mainly, since self-attention layers operate on deeply-stacked representations of the outputs of previous layers, and since training has a loss function that doesn't make the updates work in the way the article says, the analogy to memory and lookup tables breaks down quite strongly. IMO, thinking of Q, K and V as memory or key-value lookup is thus not actually very helpful for understanding what these layers are really doing or why they work. Others agree (e.g. https://news.ycombinator.com/item?id=46535412), so don't feel bad if you think there is something hand-wavey about the lookup-table and memory explanations - it is only a weak analogy.
This has come up on HN before, and probably it is most accurate (but less intuitive, sadly) to see self-attention as a generalization of kernel methods (https://news.ycombinator.com/item?id=46535549). Broadly, self-attention is just one of the simplest ways to model similarities / correlations between all inputs (i.e. kernel method), but there are many, many other attention mechanisms. There are also many other ways to implement memory (e.g. LSTMs), and many other architectures that might in fact have even more memory capacity than transformers.
That being said, this is still truly an excellent article and it really provides some great intuitions for why transformers might work as well as they do. The softmax / exponentiation here functioning to increase memorization capacity by suppressing weak activations is super clear and interesting, and I really appreciated it. I suspect is not really be what is happening or is actually important about it, given other research (https://arxiv.org/html/2410.18613v1), and because much of what works in DL is usually more about matrix conditioning and keeping gradients well-behaved generally, rather than what the actual bare final math / weights in isolation from training would tend to suggest. But it does give a great intuition for why a trained transformer can have the capacity it does.
"Here are the terms we’ll use throughout. A memory store is a function $f$ that holds $M$ memories, for $i = 1, \ldots, M$. A memory is one key-value pair: a key $k_i \in \reals^{d_k}$, used for lookup based on similarity, and a value $v_i \in \reals^{d_v}$, which is what we want to retrieve. To use $f$, we query it with an input $x$, and it recalls an output $f(x)$. A good memory store recalls $v_i$ when we query it with $k_i$, or with another vector similar to $k_i$. In the literature, a memory store that recalls by similarity is called an associative memory."
And thought "I need to disable noscript because I get crappy page view". I disabled and it didn't help. It's a little sad that there is no explanation in the article what all that dollar signs mean. I feel excluded
self attention is the best way to suppress interferences?
Also make sure to understand how self-attention layers are actually applied and the exact math and matrices involved. E.g. in PyTorch you are going to be doing sa(x, x, x), where x is the previous layer's output (essentially, embedded tokenization), and sa() does quite a few matrix multiplications. In cross-attention, you are doing sa(x, x, y), where y comes from a different part of the architecture.
Basically, read broadly here and don't get stuck on a single analogy / intuition for what these layers are doing. Focus on learning the actual math of the layer and the network first, and then look for analogies that help pump your intuition. But don't try too hard to force a single analogy onto the math.
EDIT: Also modern AI can provide useful insight into posts like this, I highly suggest using them when learning these things. E.g. https://chatgpt.com/share/6ad3aa3d-d110-83ea-a8c5-8653b34743...
Aside: I feel like this could be another intro example for students on how exponentials are a limit of a sequence of polynomials. We usually first see this in compound interest [1]. But I remember having a similar ah-ha moment when I first saw matrix exponentiation.
[1]: https://en.wikipedia.org/wiki/Compound_interest#/media/File:...
If you think that is crazy, ohh boy, just wait til you see all the other papers underlying LLMs, AI, and optimization in general.
I appreciate this post because starting from linear algebra is the correct way to form an intuition of the higher dimensional semantic spaces we're dealing with.
Embarrassing to complain about notation so simple, and, frankly, if you can't understand math concepts and notation this basic, you simply can't meaningfully understand modern neural networks: without the math, all you are doing is substituting loose analogies and metaphor for understanding. This is clearly an article for people that want more than that.
E for expectation, blackboard bold, double-bar norms and indexed max/min are all early to mid 20th century notation, making them thoroughly modern. Only summation notation dates since the eighteenth century, and its relative stability over that period is in fact rather a strength and not a weakness. Summation notation is also nearly universal in any quantitative field, and far, far more concise than the equivalent looping C-style pseudocode or functional-style pseudocode.
> You're also connecting the syntax to understanding of the concepts which is the cherry on top of the math snobbery in this thread
Because that is in fact a big part of how it works. Good notation takes a lot of time and thought to develop, and the notation used in the post is far, far more concise and packs far more information into a tiny amount of space than pseudocode or other programming conventions. Much of math is abstraction (compacting complex ideas into specific symbols or concepts, and then coming up with simple rules to allow their manipulation), and is possible in large part only because of good notation like this. See some of Terry Tao's thoughts on the subject below.
The notation here is really also all first-year introductory notation, and incredibly simple. Most CS or computer engineering students should have seen it, and anyone that thinks they can learn anything non-trivial at all about neural networks by avoiding such basic notation is simply not a serious person at all.
EDIT: And it is a mistake to imply pseudocode is generally superior or easier to understand, because pseudocode is often too precise, and implies more structure and relations and implementation details than are actually needed or important. Meaning it actually often communicates the wrong thing compared to good notation.
https://terrytao.wordpress.com/advice-on-writing-papers/use-...
https://mathoverflow.net/questions/366070/what-are-the-benef...
NNCP: Lossless Data Compression with Neural Networks (2019)
As an example - "The optimal linear memory store" talks about whitening the key by undoing key covariance through transposing and multiplying the "K" matrix, then inverting it, then multiplying it with K again. There must be some beautiful mathematical relationship there, but I'm looking at this completely confused as to how such a thing could effectively scale the key-space such that covariant keys are widened and avoid/reduce overlap.
I took linear algebra over a decade ago in college but obviously don't remember it. I'm just surprised that such semantically interesting outcomes can come of these things, but reading this for me as someone without that deep and second-nature understanding makes the math impenetrable for me. Even if I understand what gets multiplied by what and the theoretical basis for each operation, being able to read this and make sense of the math alone is impossible for me.
1. It uses math notation. This is a software development site, wouldn't this be much more clear to this audience written as psuedo code instead of a complex series of greek and latin letters?
2. It doesn't step through a concrete example. What is hollow R raised to the (M * d_v) ? Is it a number, a set, a list, a matrix?
To use an example I understand: RAG used to seem complicated to me when only math people were talking about it in math words. For example, let's take a line from wikipedia on embeddings:
> the representation is a real-valued vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning
What made it clear was learning that generating an embedding is just a function(string) -> list[float], and RAG is just taking a list[float] for your prompt and finding other list[float] that are close to it in your dataset.
So what would make this article more clear, is step through a simplified example of putting a value into a memory store and querying that memory store, showing the actual data structures.
At the risk of stating the obvious, this is a forum where people can post arbitrary links to other sites, some of which might have audiences that might not be "the exact set of people who read Hacker News". I'm not sure you'd be able to make a compelling argument that alexwang.ai is aimed at software developers and no one else.
This article is a bit more of a grey area. It is on a subject that is deeply relevant to this site and I'm sure some HN readers will get value from it. My point is that an even better submission to HN would have been an article on the same topic but written for programmers rather than mathematicians.
Perhaps, but HN apparently is pretty hostile to people explaining why something is not clear and how it could be more clear based on all the downvotes. I'll keep that in mind: "If you don't understand something, shut up!"
Clear explanations are a good thing, but different explanations help people in different ways. Your explanation seems helpful to me, and reminds me of Ben Affleck's explanation in the GQ interview. But Ben Affleck connected it directly to arrays of RGB pixel data, which clicks more with me personally, but probably wouldn't with everyone. It certainly didn't make things any clearer to the interviewer.
https://news.ycombinator.com/hackernews.html
"... the thing that makes hackers such good startup founders is that they care about more than business. They have intellectual curiosity driving them as well as the desire to make money. So the way to make a news site for startup founders is to make it be about more than just startups. ... If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity."
My thought was to show some grace, considering that not everyone's been around here for 20 years or knows that "hacker" doesn't apply only to computer programmers. That was my thinking anyway.
Imagine a nursing home that serves steak every day. One resident says "this food isn't helpful to me because I no longer have teeth to chew it. Wouldn't this be better for everyone if we had soup?". They're not suggesting outlawing steak, after all some people will get value from it. They suggesting a way to better serve the entire population of the nursing home.
I disagree with your analogy fitting as well; it's more like if someone who was just in a regular restaurant who couldn't chew said that, because they mistook the restaurant as catering specifically for the elderly just because there happened to be elderly people who ate there a lot.
Looking at the HN front page right now, of the 30 posts, 19/30 (63%) are about software or software companies. I think "this is software development site" is a fair statement to make. Saying "this is a seafood restaurant" doesn't imply they exclusively serve seafood.
Also not an issue: the quality of the article.
Expect to be challenged by content here. Everybody is from time to time. Asking for a mathy analysis to drop the math, fundamentally misunderstands the value of the math.
HN is a site for programmers (or specifically "hackers"). Posting an article to a site for programmers in math notation is like posting a French article to an English message board. Some people will be able to read it with varying levels of difficulty, but wouldn't it be better to post an English version of the same information so everyone can read it easily?
As an isolated wish, your ideas might be reasonable.
But framing them as necessary HN values, as a critique, as ideas others should conform to, you are not being sensible or reasonable.
> If you don't understand something
Ask, but if you advance weak ideas be prepared to be hit - and be glad about it, you are advised to revise the weak ideas.
Got it. Stay classy HN.
They're the same thing on an internet forum. There is no space to move through, so while in the physical world "move on" is different from "shut up", on an online forum they both mean exactly the same thing: "don't say anything".
And yes, this is stressing me out. I took the time and effort to think about and explain how this post could reach a wider audience on hackernews, citing an example from my own learning experience, and nearly everyone has attacked me for it. The original article was clearly meant to explain/educate. You'd think "here's how you can do that for more people" would be useful feedback.
The thing is, there are fundamental issues with your post. You said, "if I posted to HN a link to a really good cookie recipe, it wouldn't be a good fit for this site", which is wrong; Hackers might be interested in cookies. The site guidelines state,
On-Topic: Anything that good hackers would find interesting. That includes more than hacking and startups. If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity.
Off-Topic: Most stories about politics, or crime, or sports, or celebrities, unless they're evidence of some interesting new phenomenon. If they'd cover it on TV news, it's probably off-topic.
You also criticized the article for... using math notation... and wrongly assumed that other engineers on this site who also are interested in mathematics, like Alex Wang, would rather dispense with the compact notation and instead deliver pseudocode. This is entirely the author's prerogative, just as it's your prerogative to skip it and move on, or even criticize it. But you cannot get upset when your criticism is then itself critically evaluated, or claim others are being hostile for examining the content of your comment just as you did the content of the article. I know you weren't being hostile in your criticism, and I'm not aiming to be hostile in mine.
> You'd think "here's how you can do that for more people" would be useful feedback.
Sure! What this mostly comes down to is your misunderstanding of the breadth of our community, and the site guidelines. You truly did mean well, and I also mean well here, even if I am being blunt. The rest of your comment was fine and I don't take issue with the general sentiment.
> What this mostly comes down to is your misunderstanding of the breadth of our community
Actually I think it's the people responding to him that aren't being sufficiently careful to read what he actually wrote and to understand the message he was attempting to convey. Granted he could have done a better job communicating (case in point, this very conversation) but that's often going to be the case.
It seems like people are interpreting it in a hostile manner when from my perspective the core of it is both actionable and reasonable feedback for anyone attempting to explain a technical topic. Both of the original points (using mathematical notation and failing to step through concrete examples) are each things that will alienate a decent chunk of any potential audience. In fact the former is one of my primary complaints about wikipedia articles on math adjacent topics.
That is really your interpretation only - and it directly clashes with the explicitly intended: "intellectually curious" (which will be, if not polymaths, at least open).
Of course an intellectual curiosity space hosted around the world of startup enterprises will attract membership of people with experience in engineering.
If you check the submissions, it should be immediately evident that there is no closure.
Going to have to disagree with you here. That would very much depend on the particulars of and circumstances surrounding the recipe. This is a site for anything that would interest the typical hacker (whatever that actually is) after all.
If I consider the hacker news audience as "smart people" and/or "people with curiosity about science and technology", a couple of equations are exactly what should appear in every single post.
"I would like to understand this information, but the way it is presented is unfamiliar to me. If we had the exact same information written in a different way, I would be able to understand it."
"You are the example of anti-intellectualism that is allowing the world being controlled by dumbed down mass media"
That is some reddit-level toxicity and hyperbole.
Really.
It has been spammed all over for the last 15 years.
I just do my part to discourage it.
>It's easy to forget that the average person probably only knows the formulas for Olivine and one or two Feldspars
That being said the llm is indeed the universal translator — the babelfish supreme — and is excellent at translating whatever you want into pseudocode in your language of choice.
This is pretty approachable maths, and explain really well. I've seen far worse in programming language theory papers, which would qualify as "software development" in my mind. There are plenty of users of this forum that would find this paper useful and interesting, with sufficient math background to read it to boot.
Nice Freudian slip. It's a blog post, not a paper. But it sure reads like a paper with its priority for reading speed over explanations that peers would find unnecessary. Except it's not written for peers. It adresses "You" the average reader. Unless "You" should mean "You, the person who took, cared about, and remembered various math classes in college," the article is good old clickbait.
You should give it a go.
1. The notation used
2. The lack of concrete examples
Well putting this article in math notation without defining any of the data structures behind it, that's not like putting the article in French, it's like putting the article in Japanese. I can't understand any of it. Some Japanese speakers on this site are saying "I don't understand the problem, just learn Japanese." which is pretty much the least constructive comment they could make.
They might not all be a good read for you in particular; the choice is yours to make an effort, or "move on" as someone else suggested. You're being downvoted because you are making it sound like you are entitled to an easier explanation, an hypothetical version of the article that caters to you in particular. There are many ways you could have framed your comment differently without that baggage.
To be clear, I have no problem with math people conveying math stuff with math notation but it seemed like the type of article which, even if it will get into notation, first at least explains "the main idea of it" without notation, then expands with notation as necessary when fleshing details or implementation.
Fortunately, LLMs can work wonders in adapting content to be more accessible to a wide variety of audiences. Just give it a link and feed your exact post to it. It will be able to translate it into pseudocode and provide examples.
https://chromewebstore.google.com/detail/enjakhegoldmloghmce...