That's very cool project.
I read this blog post a few days ago about one way to look at memory encoding in models https://alexwang.ai/posts/associative-memory/ that you might find relevant
Here's the HN discussion of Alex Wang's article: https://news.ycombinator.com/item?id=50122878
Also, I really appreciate the excellent detailed writeup on your project. Curious about the mistake correction capabilities, it's fascinating, did you try breaking it up in 2 models? one that generates one that repairs? Just wondering if would make any difference in results, which could be impactful given your observation that "was one of the more effective training changes"
You could have invented transformers from memory - https://news.ycombinator.com/item?id=50122878 - Oct 2026 (149 comments)