Hacker News Reader

Comment by ketothekingdom | Hacker News Reader

Hey author here! I have a short summary here if people find it helpful: https://x.com/heyyalexwang/status/2111158818730397932?s=46

View on HN

This is one of the most well written blog posts I've seen in a while. Thanks

Thank you! It's been sitting in my head for so long. I'm glad I finally got to share it with people

It was a great read. I've been recording a lot of the ideas I had, and especially after discussions with people. Sadly never found time to sit down, digest them and write them down yet. Your article is quite inspiring!

Thank you for putting that excellent article together

The concepts of fidelity/recall and the images/animation at the top reminded me of this small playful project that touches upon these topics: https://nicobrenner.github.io/lossy-git/

That's such a neat idea! Now I want to try it out too

I find some aspects of your writeup to be... quite annoying if I'm honest. The writing is good, but you fail to define several of your terms early on.

What is d_k - I assume it's the dimension of the k vector. Similarly, I assume d_v is the dimension of the v vector. What is the "i" suffix for here at this early stage? It doesn't need to be there, it's just confusing.

You say the keys are "used for lookup based on similarity". What does this mean? Similarity to what? The existing keys? Something else?

Your first example then introduces the identifier "M", which seems to be the number of memories. Again, this is not defined. It speaks again of similarities, and introduces "weights". But the matrix Mxd_k is a matrix made of all your keys, which aren't "weights" - or at least this term is new. The term "tokens" makes an appearance around this time also - something else that needs defining.

I think you need to go through the text of this article, with an eye to someone who is reading it for the first time. Define every term you use, remove unnecessary notation - and also perhaps explain at the start why one would be interested in a "memory store" of this nature.

There is a place for something that really explains what's going on here. I've yet to find it.

I found many of the things are clearly defined after the sentence „Here are the terms we’ll use throughout.“ right at the beginning (e.g. M, d_k, d_v).

Where he does NOT state what d_k or R^d_k mean and there are countless other examples. I happen to know because of classes I took, but it's insane and obnoxious to assume that every reader did or went through any higher education at all. This is not high school math. It's not terribly complicated either, making this situation even more frustrating. The author could have spent just a little extra effort explaining these things to multiply his target audience - but he's either incapable to think from a different POV or pandering to his peers by not wasting time defining "the obvious". Pure academic blindness on full display here.

Instead of being condescending and annoying yourself, did you even try to reference other texts or resources to try to figure those things out? This is a freshman's complaint

Agreed 100%. Mathematicians SUCK at writing. No matter how hard they try, they'll gloss over important non-trivial definitions. And I think this author has been trying... but seems to at the same time ignore the negative feedback like yet another academia snob.

https://www.nature.com/articles/s41586-024-08548-w

and others. Hippocampal sparse coding/orthoganalization/pattern separation and completion/etc may be interesting search terms for you, if you are unfamiliar. If you do happen to be a hippocampal nut like I used to be during grad school, I salute you

Really awesome write-up, and the kea is adorable!

I have only passing familiarity with how LLMs work and I found it enlighting. It makes me curious how recall quality degrades when quantization is applied. It looks like it’d gets worse quickly.

Great explanation, thanks! Would be cool to see this for people BSky instead of ex-Twitter

Fantastic blog post.

Curious to hear more about the stack for your blog? Thanks

I use a combination of Hugo with obsidian-style syntax for everything

No, after many years, I still hate Twitter/X.

Please write a proper Blog post.

> Please write a proper Blog post.

The Twitter thread is a summary of the blog post.

You can just read the blog post.

Yeah, found it. I'm reading it now.

I'm not sure who the "you" in the article is, but I find both the article and the summary void of any attempts to explain anything intuitively, extremely math-dense, and essentially incomprehensible.

by far this is the least math-dense and most intuitive explanation of these ideas that I have seen committed to print anywhere. consider that you might need to do the work of developing improved personal understanding, rather than complaining the expositor hasn't done the work of getting to the conceptual essence. sometimes the conceptual essence is mathematical, cannot be further compressed, and that's okay. I struggle to see how this blog post could possibly be "simplified" beyond defining a few terms more explicitly.

Precisely, the mathematical insight is the essence of the post here, you can't simplify it or change it to some other form without destroying the point and the beauty of the insight. And yes, the math here is trivial. The hubris, anti-intellectualism, and lack of curiosity of these complaints, on HN of all places, is really shocking.

Love how the comment below yours states it's one of the most well written blog posts they've read in a while. Classic HN, full of highly opinionated folks.

Reply on news.ycombinator.com