Memorilla: Latent Semantic Memory for LLMs | Hacker News Reader

Memorilla: Latent Semantic Memory for LLMs

arxiv.org

123 pointshandfuloflight13 comments

So if I'm reading this right, it's basically a specialized embedding of a RAG, and then a model in the middle takes the agent's queries and returns responses in a form that is injecting into the main model's input token stream directly i.e. skipping actual text. And the injection format is INCREDIBLY token efficient. Sort of injecting a vibe rather than quotable text. Their results of "good on concepts, less good on quotes/exact text retrieval" seem to intuitively line up with that.

It looks like another difference with RAGs is that instead of reranking the doc vectors and taking the top 5 or so, this paper mixes all the docs (weighted by relevance) into a fixed number of "memory tokens".

It looks like this helps avoid a problem I've had with RAG - when a fact isn't local to a single patch of text.

    > ...a problem I've had with RAG - when a fact isn't local to a single patch of text

Is this a problem, though? Vector similarity should surface the relevant, related patches.

But only in documents that make it through RAG’s hard cutoff. There’s an explanation of this on the related Stanford site at https://memory.stanford.edu/ (see “How MEMORILLA differs from RAG”)

I think its offering a compromise between accuracy and tokens. You don't have to use as many tokens and you get pretty good accuracy, but not the absolute best. I think, however, in many situations people aren't super concerned about the tokens as they are with the accuracy.

I was just trying to find work about this yesterday.. Wondering, if we have multimodal image,text,sound models, why not multimodal, text,graph models that can store their memory in more efficent format than text in their context, without needed external tooling..

agree

I am finding that semantic ~= lexical when you have something like a frontier model driving. Using full text search engines is still the most compelling approach for handling large document sets.

I really struggle with the semantic / embedding concerns when we have a generative model that can reverse engineer a Mario 64 rom dump to run on my toaster oven. These models know how to compose Lucene & other FTS query strings in such a way that we are effectively getting semantic search over purely lexical systems.

I think semantic embeddings are only a part of the picture. A hybrid system is definitely needed but only text based approaches are miserable. My agents keep on repeating the same mistakes across sessions. We need to first get rid of this notion of a session.

This is true, but consider that we have agentic token injection (what you are referring, prompting), embedding injection (memorilla) and continual learning.

So the first will update _what_ information the model reasons/infers about, the last will update _how_ the model reasons/ infers about.

These are very different.

As far as I understand memorilla sits in the middle. And it embeds not tokens but embedding vectors with all their complexity. The compression (semantic) is significant and in my mind is super interesting.

The types of systems from the paper get more interesting when you’re dealing with really tight voice-based conversation loops. If the workload isn’t latency sensitive and you can just have a harness execute the types of searches you mentioned it’s rarely worth it to explore these types of optimizations or memory structures (or using embeddings at all). However, if I ask a model for something we talked about yesterday I don’t want to have to wait for it to reason and then make a tool call then read it and reason again. Even the, “let me check on that” gets tired quickly. They mentioned a 27x speed up putting the retrieval at 0.79 seconds vs 19s which is very promising for voice-based personal assistants.

The speed-up has another component: semantic compression: you express more in a single embedding vector vs. tokens retrieved.

Your point is valid for the agentic tool search, i am just saying there are two things lurking.

memory as a modality is such a cool idea!

Reply on news.ycombinator.com