Agent memory needs its OpenTelemetry moment | Hacker News Reader

Agent memory needs its OpenTelemetry moment

lorekit.io

171 pointsmthines103 comments

Is this a significant problem? It seems like there are 15 solutions to every AI “text format” problem, but all are merely structure over feeding the AI informal English, structure which I believe the “transformer” model is specifically able to strip to a common form (which is how training it on the entire internet worked).

AI does benefit from learning to use skills, and being instructed with good structures (like “save [specific important kind of detail] in memories” and “plan before implementing anything”). But none of these memory solutions seem better than any other, they’re basically just different folders. Your CLAUDE.md can say “see AGENTS.md” and AGENTS.md can say “write memories to .agent/memories/“, and I’d be interested in a measurement, but I assume it performs almost the exact same as native CLAUDE.md and .claude/memories.

One thing I’ve noticed AI does much better than me, is not have OCD over the “perfect” solution.

As someone not involved in AI development, it seems really surprising that memories are another text-format issue. Seems like (in a human analogy) something more equivalent to your notebook or scratchpad.

Memories are lossy things that live in our brains, I’d expect they to be stored in some native LLM encoded representation (which would make the lack of standardization understandable).

This probably seems like a dumb distinction, but I think some of the problems in the article come from calling these things memories. Like, the prompt injection/provenience issue. The model has received some notes from someone about a topic, if it they were treated as “notes from <whoever sent them> about <topic>” instead of “my memories about <topic>” that should put things on the right footing to start at least, right?

The issue with storing the memories in some encoded representation (e.g. in embedding space) is that this would turn that representation into something that has to be supported "forever". I have projects that I have worked on with Opus 4.6, 4.7, 4.8, 5, 5.5, as well as Fable 5 and 5.1, and obvious some Sonnet subagents along the way. And I expect to keep working on the same codebase when Opus 6 and Opus 7 release. Text is trivially compatible with all of those, but their embeddings all differ. And whatever other representation you come up with right now in Opus 5.5 might no longer make sense in the model architecture of Opus 7

Imho the interesting challenges with memory are about "what does this memory belong to" (attaching it to lines of code or similar), "what does it relate to" (webs of memories), "is this even still true" ("sleeping", "reconciliation", etc). But there's no real consensus what is even useful to handle, never mind how to handle it

This is what happens when you treat the LLM as a black box, you end up building the same stuff around it. Like trying to make your dog understand your intentions. You can train it to do tricks and it recognizes specific reward patterns but most dogs have no idea what you’re saying beyond basic triggers.

It's sad what software engineering has been reduced to for a lot of people. It's basically pickup artists compiling a list of "body languages" for LLMs, pretending as if there's science behind any of their interpretations.

At least body language has some studies put into understanding it.

That may be so, but most "body language tips" circulated online are cringey and lack the rigor and nuance of proper scientific studies. I think of human reactions as more complex than what those kinds of online materials imply.

It's even worse when applied to LLMs, which are more unpredictable.

At the very least the pick up artists have a common baseline in that they are also human.

And LLMs have a common baseline in that they produce tokens. The comparison is apt. Both fields should be studied. Body language and Prompt Language.

People had this grand idea of software engineering and how much complex it is and problem solving and this and that and you actually deserve 250k to do your “complex” job.

In reality many of them have not needed to look at a data structure or do any real computer science for years. They were just attending bullshit meetings and approving some pr with some basic sql query or some new react code.

Common people knew very little about computers and so they thought they were running on magic and that the software engineers were the magicians.

Can't we just appreciate computer science, or any other field of study, for what it is? It's unfortunate that so many people now treat the fundamentals underpinning our whole profession as a nuisance to be left for LLMs, or resume padding.

That's what 'democratization' looks like. Everybody's an AI engineer now!

Honestly I have just a few lines in claude.md. And one of them is treat agents.md as the main file and update skills in the .agents folder and use it as source of your skills. Its working so far.

Entire blog post reeks of LLM. If there was an adequately reliable way of detecting LLM text i'd build a browser extension that would just filter it out altogether.

Thought the same thing and was surprised how high this was ranking. I wonder if we are starting to look past the LLM stank and into the core of the idea (which is solid in this case)? If HN, a highly discerning crowd, is able to look past it then many other audiences are probably okay with it too

Often people don't even read the article, they just see a headline they agree with and upvote it.

I usually read the comments before deciding if the article is worth my time.

HN isn’t that discerning, lots of LLM-written posts seem to zoom up the feed (especially if they have a good headline). I’m not sure if it is a bad things really; often it feels like the posters here ignore the contents of the link, but come up with legitimately interesting discussions kicked off by the headline.

Agree about the mechanics.

It’s a bad thing if you valued the thoughtful, insightful, well-structured technical writing that used to be behind many highly-upvoted hn links.

Multiple barely coherent LLM-authored technical rambles with catchy titles have been hitting the hn front page daily for the better part of a year.

Exactly, and LLMs cannot describe architectures, period. On the surface it looks coherent, but everything is vague and lacks rigor. Instead of defining concepts and giving them a name, LLMs keep referring to them using vague terms such as "address and key is where the memory is true". What the fuck does that mean? Give each concept a name and explain them.

This has no value because noone can separate what was designed by a human from design details that the LLM made up on the spot. That's fine for one-off generated stuff like web pages, but as a proposed standard for everyone to build on? No.

It's always doors and seams in my case. Ups, downs, peers, whatever but actual terminology used in the spec. It also has the writing guide with examples of good and bad, but it always needs a second pass to actually follow it.

Have you tried Pangram? It’s quite reliable for the use case you describe.

People here give AI detection a lot of flack because it’s problematic if your’re using it, for example, to evaluate student essays and make accusations of AI plagiarism.

But if you just need an occasional sanity check while browsing the web? I highly recommend.

I am a human pangram at this point, it's LLM.

> If there was an adequately reliable way of detecting LLM text

Yes it is and it's not a technical one: https://digital-strategy.ec.europa.eu/en/policies/eu-icons-l...

There is no reliable way. It is like saying that LLMs can only produce output that is tightly constrained and user cannot control it to go outside of these constrains. If that would be true, these companies could not even dream 1 trillion valuations.

If user bothered to control the output enough that it doesn't reek of LLM it's okay to not mark it ass llm vomit.

The question is how do you detect those who do not comply?

To point out the completely obvious, the approach described in your link requires compliance on the part of the author.

Given that the author could already have just inserted the text “hey I generated this via AI”, I see absolutely no reason why they’d go to the effort of adding an “AI label”.

As an european, this screams classic EU move for me. It adds bureaucracy for the good actors and do nothing to the bad ones.

That would be great. Or, for the physical world, combining some kind of smart glasses with the new Pangram image detection which seems to work well on rephotographed data. Like the _They Live_ glasses.

At what point we just start hating LLM because it is LLM?

Honestly I do not care if article was written letter by letter by human or written by agent steered by human.

I care about knowledge and quality, it it is there then it doesn't matter how was it written. I too hate LLM slop but I understand that not everything done by LLM have to be slop.

At the moment it's a good proxy for quality and intention. IF it was mostly generated, it's probably mostly meandering bullshit which doesn't quite make sense.

Yes the style is annoying and bombastic, but the objection goes deeper than that; it's annoying to have someone generate text with an LLM from a small outline or prompt, not even edit it for sense, and dump it on readers.

I think many people are just fed up with the specific voice LLM's tend to write in. Doesn't help many of us now live in a world where most of what we do is read LLM prose in claude code and what not

>At what point we just start hating LLM because it is LLM?

At the point where you read enough LLM written text with zero substance narrating wikipedia article or heard that voice on youtube restating the same thesis over and over for 15 minutes.

The rule of thumb: if I can detect it's an LLM without trying to detects it's an LLM and the feeling doesn't go away, it's a slop.

Hey, author here. I wrote the blog post and of course got assistance by AI to rephrase sections, gather data and do comparison, and make graphics - also because I'm no native english speaker and wanted to improve the writing.

The sections, points, ideas are by me :)

Fair criticism. To be precise: I dictated the outline, specific sections and ideas, then used Claude to draft and rephrase sections and iterated on it with Claude. I didn't think it mattered, as the points are mine - but I can see people care so I should have said so in the post. :)

I will take this into account for future posts.

As a native English speaker, please believe me that it doesn’t read well. It reads like Claude or another LLM wrote it. Not only is it Claude’s voice, but the outline of ideas is hard to understand. It’s not a good idea to generate text at a level above your mastery of the language because you won’t know what specific feedback to give. (Similar to the problem with vibe coding and then trying to make it production grade.)

No, you didn't. Claude wrote the entire article and the entire website.

Pangram: 97% of this text is AI.

You are lying.

I imagine you probably wrote an outline with the help of Claude on this topic, then got it to generate the article including illustrations based on that, gave it a look over then pressed publish? Did you write any substantial parts of this yourself?

I would rather have an extension that blocks all the HN comments that complain about a post reeking of LLM. And probably also posts like this that complain about the complaints about the complaints...

Hopefully there's also going to be a browser extension to filter out people complaining about LLMs.

And filter our LLM slop and sock puppet replies. Looking forward to the clean handful of comments left

I'd say it's more annoying to encounter HN comments that only say "this is AI" than the AI content itself. This article addresses a very important pain point and is trying to approach it at the right level. Of course the authorship is LLM-assisted.

If only there were a filter that could detect these kinds of comments and collapse the thread.

This is such a thought terminating cliche at this point. Is it possible for you to comment on the content of the article rather than just informing us you didn't like the format? It's not helpful

Four different ways, five approaches. Three statements and nine topics .... Every ai slop text has this endless useless crap.

Hey, you either write blogs using LLMs or you get steamrolled by people that do. /s

On the general premise, I agree. But I don't think that moment is today, not yet anyway. I don't think we understand the right way to approach context and memory yet or how long they should last for, never mind standardizing on an abstraction prematurely that ends up being like SOAP or SGML.

What's better than SGML? What would XML an HTML look like if they were based on that instead?

HTML is SGML. You know that while doctype things you're supposed to out at the start of the document? That specifies the document schema of HTML at a specific version for the current SGML doc.

Yeah, hence my question. What would HTML and XML look like if they were based on this thing that's apparently superior to SGML instead of being based on SGML?

Fully agree. Any solution people think they have today is working on assumptions that are probably not correct. And things are still moving very fast.

The need for a unified format or spec is greatly reduced in the case of Agent memory because by definition it is read by agents, and agents are already instructed / programmed to read memory files of different formats. Even if such instructions did not exist, agents are extremely good at traversing files to get the necessary context it needs.

This is true and it probably is actually fine, but in a way you don't want it to be wasting its reasoning abilities and context window on figuring out which text files to read.

It would be like trying to accomplish something as a human but all your reference material is always in a different format, it would be maddening.

> Agent memory because by definition it is read by agents, and agents are already instructed / programmed to read memory files of different formats.

Only it's not read by agents, and largely ignored.

From Anthropic's recent self-congratulatory article:

---start quote ---

Injected memories use up valuable tokens, and an incorrect note can be carried into every session. Loading a note also doesn’t guarantee that Claude acts on it: our documentation on Claude Code warns that longer instruction files reduce adherence

...

It loads only the first 200 lines or 25 KB of MEMORY.md

https://www.anthropic.com/research/components-of-agent-memor...

--- end quote ---

Do we really need this..? What LLMs do well is interpret natural language. So it's a sponge for whatever format is used in a specific project: a predefined format is not really needed.

Besides, do we really want memory of user A's agents to co-exist with memory of user B? Not really. So the argument for collaboration and sharing is quite slim, at least for coding agents. The repository specific context can be encoded into a docs/ or architecture/ directory, just like it's been done pre-agent times as well.

For monitoring deployed agents: maybe. For this I built (yet another) memory management tool slivingdoc.dev, with this it's possible to pull down the durable context of all agents. But here the problem is a distributed system one, not format of the memories.

Structured metadata like this is for the non-LLM tooling layer, primarily your harness, so (eg.) it can interpret multiple formats without needing to burn tokens every time.

Premature optimization imo

I'm happy with the otel LLM extensions, and would rather this to be a meta protocol vs actual protocol/standard:

1. Harness memory rapidly evolves approach across the full stack, from data storage & rep to overall concept. Arguably we are in an early primitive era because text is being used instead of semantic vectors etc.

2. Even something as basic like a knowledge graph has a wide diversity of approaches.

3. Similar to REST APIs, what I would argue for is the requirement of vendors to enable certain bulk actions. But at a meta level, not a protocol.

Speaking as a profitable harness designer (Louie.ai) who was messing with autowikis almost 2 years ago, this stuff is closer to the beginning vs end, so feels distracting. Otel+lmm extensions are a diff story for what our users need plug and play. A skill file for our memory systems is enough for a2a use.

It's too early to advocate for standardization for this problem. Otel developed after decades of experience in the field and we explored countless solutions and drew from them when developing Otel.

Saying we've explored enough in agent memory solutions right now enough to standardize it and add friction to any further experimentations would be an error I think.

I am curious about what people do these days and how do they tackle this problem. At repo level you can alter by with AGENTS.md and perhaps a curated set of additional docs with the project history, but as soon as you get out of it and want to have additional context the solutions available are too many - Hindsight, Graphiti, and many others; or simply roll out your own? I would like something I can spin up with Postgres or with an S3 bucket and that writes me .md files while retaining some sort of relationship among different entities - however it looks like there is no “simple” solution out there yet, with many of the existing implementations being way too elaborate and/or requiring exotic DB solutions. Would be very interesting to hear if someone knows of a bare bones solution I can plug into any agent.

The simplest I’ve found is to add instructions to the AGENTS.md telling it to keep a knowledge base in a directory and refer to it. You could add a hook or something to use the AWS CLI to sync the dir to S3.

The agent actually did pretty well managing and searching it with grep.

A lot of these memory systems are reaching for the stars and I sort of suspect we’re way past the Pareto point for coding there.

Right, I have a simple key-value thingy backed by the filesystem. Not even gzipped. Maybe I will get something elastic shaped or what not at some point, but likely not.

Like what are you store there that text files aren't enough.

Why not a simple git you can push to main directly?

The notes and code are usually tied together, and putting them in separate repos just forces you to sync them.

The notes for v1 probably don’t apply to v2.

Maybe different if you’re doing a global memory instead of a per-project memory.

No it doesn’t. There’s still a lot of innovation / change happening in the LLM space, and no solution is one size fits all. AFAICT, solutions have to trade off across 3 primary concerns: speed, accuracy, and cost. I don’t even think memory as files is even set in stone yet. There’s so much innovation and progress to be made in this space.

Agreed. “Memory” isn’t even a single problem, let alone one with a single optimal solution.

Its a solution trying to find a problem. Memory, and in-fact entire storage layer needs to be invisible for agents and daily workloads. Nobody is building/selling agents for their amazing storage, but what can they do with it.

Nobody is building agents for memory, but they are building on top of it. you make solutions simplify themselves so that users can focus on actual business problems, nothing more.

Small plug- I have been running https://github.com/Polign/recall in a few production workflows, and most I hear about is, that it just works.

What about .. reference identity.. currently memories and recall are referencing local things, that are specific to the environment they were made. and in ways that are as context eficient as possible like @message100Line10-50 or sessionsabc@100-300 is ./yourfiles/thisfile.mb ..

but is were going to make the memorys portable.. we need to make the things they are pointing to portable also..

This is interesting but also think about how we as humans memorize things in code

I don’t think “oh it’s lines 10-50 in file A that has this code I’m looking for”. It’s much looser, something like:

- maybe I can’t fully remember file A’s name but I can fuzzy search for it based on what I do remember

- I search for a function or variable name that I remember, again kind of fuzzy searching. Or if I can’t find it,

- I start quickly scanning the whole file for relevant variable/function names

Obviously storing exact references will be more efficient at lookup time but to your great point, the environment is changing too much, especially now. Having just a) a starting point on where and what to search and b) an understanding the code base will and should change , feel like some of the principles a memory system should assume. Not that it’s mutually exclusive with what you said - just not sure the juice is worth the squeeze to make the thing memory is pointing to portable.

I guess it depends of the use case. For episodic memory that need hard evidence, like .. what did we decide last week.. it would be good to have the link to the original document to look, rather than just an extracted text.. Especially if those memories are going to be subject to reranking or "dreaming" or some non-deterministic clean up .

I have been running openviking self-hosted for 6+ months now and it has worked well for me. It is open source, self-hostable, and you can bring your own LLM and embedding model.

The bytedance team makes it. I often wonder if it does not get attention here in America b/c it is Chinese?

Well, coding agents are smart, they can use all kinds of memory. A simple skill on how and when to access it is enough...

Should we even be storing 'memories' in text files at all?

Imagine how humans would function if we carried our memories around in overflowing notepads and had to consult them every time we wanted to know a fact or remember a person we had met, it would be exhausting and inefficient.

Individuals don't store memories as language, but language is the unit of portability between individuals.

I'm skeptical that there's a medium better than language that's portable between models. Would love to be proven wrong since the solution would likely be fascinating.

Any system where a memory written by an opus orchestrator isn't available to a sonnet implementor doesn't seem particularly useful to me, so portability is important.

It would be nice if local models will unlock the way to put memories back into the model itself somehow, but it's not big labs who would bake something useful here I'm afraid.

How does this handle parallel sessions writing the same memory? From a quick look at the repo, it seems that one session can silently overwrite another. Anthropic's recent memory post shows how severe this can be (https://www.anthropic.com/research/components-of-agent-memor...). Should the spec cover this?

Maybe don't give all your agents the same memory. Shared stuff can be in the repo they work on as files.

I agree this is very much needed but I doubt the other ai companies would a-comply with this as they need users and mainly paid users otherwise they'll lose themselves. What I think is that with 20$ subs of 2-3 ai can give you more usage compared to their 100$ subs if you do it that way. I don't know will these companies work together to form a standardized memory system. I highly doubt that.

I think It needs to be as diverse as people .. If agents are to become our constant "muse" like companions, I want the wiring of the brain of mine to be as wired as possible.. Why not juggling chain saws in orbit.. https://www.youtube.com/watch?v=JcJSW7Rprio&t=17s

I'm ready for the conversation to just assume a linux file system with mostly markdown. And you schedule some time every day for an agent to make it not suck as much as a living archive.

What are some real issues now? Auth? You want to retroactively implement boundaries? Is version control the problem?

Working with LLM agents comes with a hard pill to swallow: they are already remarkably good at figuring out what to do on their own. In fact, the more rigid structures we impose on a system, the less efficient these agents become.

While there have been many attempts to structure how agents operate, a recent Anthropic article showed that when they didn't tell Claude how to use memory, it actually used its memory better. Ultimately, we have to come to terms with how little is required from us when LLMs are this intelligent. often doing things better in ways that we sometimes cannot comprehend.

I don't buy it. You can steer them gently to at least write human-accessible project plans—and then session compaction, agent switching, and review is much more doable. I've been iterating on this for over a year: https://photostructure.com/coding/claude-code-tpp/

at this point i'm pretty convinced agent memory is strictly harmful. it remembers the wrong things, and then turns them into an inane focus for each task. i disable it universally on every harness. if i want memory, i specifically instruct the agent to record findings into a file.

Same. Everybody builds one, and then eventually realizes it makes the model stupid and confused. The right solution is pretty much always developer documentation in the repo, in roughly the same shape you'd make developer docs for humans that you want to be able to get up to speed about your codebase really quickly.

This has the benefit of being something you can easily read over every now and then to make sure the model hasn't baked in some random comment you made about a specific circumstance a month ago and made a weird religion out of it.

Also introduces massive confirmation bias. Majorly gimps the model. I remember I was exploring hexagonal architecture on chatgpt and then future conversations with open ended architecture questions just gave me those same patterns.

“Oh damn look its recommending something that looks like hexagonal architecture. I guess hexagonal is a really good - hey wait a second …”

Agreed. Claude Code often tries to write one-off decisions and transient problems into memory. And when such things appear in the context, Claude will start doing questionable actions, because it saw something unrelated and wrong somewhere in its context

I strongly agree. This is becoming a general principle along the lines of “code comments that describe what the code does are bad.”

Memory just ensures there is a fairly high level of drift and inaccuracy in your context. Unless YOU are manually curating it.

Its similar to wiki systems where your agent produces tons of documentation and decision records, etc. Its a bunch of assertions about your codebase that you aren’t vetting and which is at most 90% correct. Thats good enough for a feature or bug fix that you can validate with tests but for the love of god throw the details away. You don’t want to codify that. It’s polluting the context and gimping the model.

This seems like local memory for one user. The need for corporate agent memory that works across users and company processes is acute.

Yes, also I doubt that we already know how to best implement memory for plans. Is it really only files? Maybe some db approach with queries is needed?

It does but it needs someone who has actually thought deeply about the problem and not someone who asked an LLM to “15 competing standards” the problem and shipped what came out. Research is still being actively done on the right shape for this, and the answer is changing under our feet as more capable models literally use memory differently than less capable ones, changing nothing but the model. See e.g.:

[0] https://www.anthropic.com/research/components-of-agent-memor...

Do you mean another overcomplicated enteprise standard teeming with constant CVEs to increase surveillance and control?

We need just a few more fully AI written articles and vibecoded startups about agent memory.

Surely the random seed of my agent session will spit out the right one.

So we need another complicated format to store information so that the machines which read unstructured text can work with it?

This is slop. If you can’t be bothered to even edit your AI slop, I won’t be bothered to read it.

Maybe I’m going to get downvoted for this but premature-standardisation is a problem.

We only have OpenTelemetry because of years of people working out what metrics and logging look like.

It’s too early in my opinion to standardise agent memory.

Not trying to be disingenuous but what's wrong with just a normal filesystem?

The best version of memory I've experienced is when I give my agent a memory directory and just let it do what it wants in there, with minimal rules on my end.

Files aren’t indexed and vectorised by default

This is true but is it proven with reasonable confidence that indexing and vectorizing the code provides better results. I would assume it would if another agent is responsible for it and your coding agent only talks to that agent. Even there I am not 100% sure. Probably not worth the effort though.

it's called a markdown file

I'm not convinced that agent memory is more good than bad.

If there is something an agent needs to know across sessions shouldn't the operator also know and include that information intentionally?

I often catch Claude (and wardon) noting information that is only pertinent to a single season as if it were a forever rule and then I need to tell it to forget.

Maybe the moment we need is to remove this capability?

Some times when I see ai generated text like this, it feels possible to see how sophisticated their AI skills are if they couldn’t even get the text to render better, and how that might transfer to the code itself.

This looked interesting, unfortunately it feels far too long for humans to spend time reading.

People might really be starting to write like LLMs, it’s why reading books is the single biggest anti-slopping protection for the human brain.

The “shape” of the article was a word generator performatively generating larger words where simpler ones would do. “That holds”

Hacker News needs its slop-rejection moment.

I decided long ago to not wait for you all to figure this out.

Index it all. Embed it all. Give your agents the ability to query it all. Eval different configurations to figure out what works best for you.

Reply on news.ycombinator.com