HNHacker News
TopNewBestAskShowJobs

kaydub

638 karma · joined April 7, 2020

submissionscomments
kaydub··on Agents don't need memory, they need documentation
I think there's a balance between the tokens burned up rediscovering things vs tokens being used keeping docs/decisions/skills/etc.

And I think a lot of places are burning way more on the latter than the former. I think the former is more cost effective due to caching as well. So I HEAVILY lean towards the former especially with how effective I think it is these days.

kaydub··on Agents don't need memory, they need documentation
I'm assuming these QA agent are automated, probably in CI/CD. And I'm assuming the prompt that runs these QA agents is from a codebase in VC. Why would I put QA instructions into the codebase's repo as a top level markdown file and not in the pertinent part of my QA process?

Even with your documentation, the llm is gonna do a lot of that grepping and discovery. Unless you have such comprehensive documentation that it's basically code itself... in which case, it should just use the code as documentation.

The good news is that I don't think what you guys are doing is going to be really bad. It's just not near optimal and it's creating this weird dogma around all these "tools"

kaydub··on Agents don't need memory, they need documentation
This is what commit messages are for.

You guys are just reinventing the wheel and it's fucking square.

kaydub··on Agents don't need memory, they need documentation
Maybe but I doubt it. We're all working on pretty similar things, that's why the LLMs work so well.

You're just stuck on your dogma.

kaydub··on Agents don't need memory, they need documentation
Wow, what a strawman.

I didn't say give no direction.

I'm saying a lot of devs/engineers little rain dances aren't really bringing the rain.

Keep a basic CLAUDE.md/AGENTS.md that keeps high level details. Have a way to reference other projects/apps/codebases to use as reference architecture.

Don't keep tons of markdown files. Don't keep "decision" documents (these are probably the worst). Don't make skills for every little thing (superpowers are dead these days, stop using them). Oh yeah, don't have the LLM generate docs for reference later... if it can generate the docs it doesn't really need them.

kaydub··on Agents don't need memory, they need documentation
Nope. Doesn't work. Not consistently. Not without overhead or maintenance.
kaydub··on Agents don't need memory, they need documentation
"docs and maps" are hardly specifying preferences. I DID say I still often keep a high level AGENTS.md or CLAUDE.md. Where I have them, they're super high-level. In places where we have well defined and followed standards that top level markdown file can be as simple as "check out this other repo for patterns" and where I don't, the markdown file is high level overview of what the app is and then a high level overview of architecture or some preferences. Always a shorter/smaller doc.

I don't mind SOME documentation. I'm growing extremely frustrated with the absolute MOUNTAIN of docs, comments, decision documents, etc being created. I'm so fucking tired of the LLM rube goldberg machines being built.

kaydub··on Agents don't need memory, they need documentation
Nope. Opus and Sol for the most part. Haven't even used Astra or Fable.

At work it's all Anthropic. I spent a while on Sonnet models but have been pretty consistent about using Opus now. For personal projects I bounce between Sol/Terra and Gemini Flash.

Most of the models are good enough now. You don't need all these documents, memory, plugins, skills, etc. MCPs are good for hooking up to external sources of data (at work we have gitlab mcp, datadog mcp, and our knowledge base mcp... which the knowledge base is mostly worthless now because of all this AI generated documentation, but I digress). I DO still use beads on a lot of projects, but not always. My prompts these days are lazy as fuck: "review this repo, check out this other repo with architectural patterns we should follow and libraries/terraform/etc we should use, <basic desc of goal>. What do you think? Let's review everything and discuss before we start building"

Oh no, I might have to stop the llm agent on occasion, or review what they did and tell them to fix a couple things. Way better use of tokens and my time than creating some rube goldberg machine. Way better than keeping a giant decision file that goes stale and causes context corruption because the llm only saw the OLD decision and not the UPDATED decision.

kaydub··on Agents don't need memory, they need documentation
Yes, you're right, I'm not providing numbers. But that's because I'm just using the tool as designed... the vanilla version. I'm using the baseline. I believe anthropic and openai publish plenty for my side of the argument already.

I'm not the one bolting things on claiming it increases productivity. Why would I ADD stuff without any proof or evidence that it works? I'm simply NOT adding these things.

kaydub··on Agents don't need memory, they need documentation
Okay, so using superpowers. Have you actually looked at the superpowers? They're garbage. That shit was maybe helpful a year ago. It's now just cluttering up space.

I think a lot of people are stuck on patterns that "used to work" (and I use that loosely because I really feel like LLMs are getting work done IN SPITE of using all these additional skills/plugins/etc). Go ahead, delete all that shit and try building without. Yeah, sometimes you may have to hand hold the LLM, but that hand holding is a LOT less than doing whatever this is.

This all just sounds like a rube goldberg machine you've made.

kaydub··on Agents don't need memory, they need documentation
Commit messages are also great for recording the "why" or other details that don't end up in code.
kaydub··on Agents don't need memory, they need documentation
Not just that they get stale, but if you're using the LLM to generate the doc you don't need it.

I don't think I've ever seen an agent review a doc and then NOT also go look at the code. So skip the middleman, just have the agent look at the code.

kaydub··on Agents don't need memory, they need documentation
I've had the LLMs undo past decisions MORE from the docs than from having no docs.

Old decisions always end up in the docs. If the LLM gets a whiff of an old decision, but doesn't get the update to that decision, well now you're doing things the old way again.

kaydub··on Agents don't need memory, they need documentation
That just sounds like a code-smell to me.
kaydub··on Agents don't need memory, they need documentation
I did say a high-level CLAUDE/AGENTS.md is fine.

Make a high level .md, let it rip, iterate. You can give specifics in your original prompt.

I don't need to document the framework or the libraries etc. Pre-code, I tell it in the prompt one time. After it inits the project it's in the code.

kaydub··on Agents don't need memory, they need documentation
Most often the "why" should be self-evident.

When it's not, there are commit messages.

Please for the love of god, quit generating markdown files (especially having the LLM generate the file, because if it could generate it, it doesn't need it), quit generating "decision" docs, and stop having more comments than code.

kaydub··on Agents don't need memory, they need documentation
High level docs are fine and can be okay or I can see them as being helpful. I did say I'll keep an AGENTS.md/CLAUDE.md. There also are SOME docs and sometimes the LLM makes the docs anyways. I'm not an absolutist, but I'm REALLY fucking tired of seeing these giant .md files, "decision" documents, more comments than code, etc.

It's a LOT of cargo-culting overly complex AI workflows. It's devs/engineers making rube goldberg machines.

Everyone is doing their own little rain dance and when it rains they say "see, I told you it works"

kaydub··on Agents don't need memory, they need documentation
This is one of the things I REALLY don't get.

If you got the LLM to generate the docs, they don't need the docs.

kaydub··on Agents don't need memory, they need documentation
Most of the "why" should be self-evident.

If it's not, that's what commit messages are for.

kaydub··on Agents don't need memory, they need documentation
The "why" should most often be self evident. If it's not, that's what commit messages are for. Not more markdown files and code comments.
kaydub··on Agents don't need memory, they need documentation
Commit messages are GREAT.

I'm mainly aiming my frustrations at all the markdown files being committed, all the additions to knowledge bases, all the comments in the code (especially the ones referencing specific JIRA tickets). This stuff isn't helpful, it gets hella stale. I've had the LLM fuck up plenty due to these docs and comments.

Commit message ARE EXACTLY where architectural decisions or nuance should go. Not another fucking .md or more comments.

kaydub··on Agents don't need memory, they need documentation
> You could say that’s an environment issue, not a code issue

Yeah, that's exactly what I'd say. Or maybe you're just approaching the problem now.

You have to prompt these QA agents, correct? Why not give the instructions on where to get credentials in the original prompt?

> But as far as actual code IME at a minimum we have to tell the agents our general repo structure and architecture patterns.

I did say I'll keep an AGENTS.md or CLAUDE.md. I keep that SUPER high level. The most depth I'll give is info about example projects (That the llm can reach using an MCP) to follow for architectural patterns.

kaydub··on Agents don't need memory, they need documentation
What are you measuring to prove this?

The docs end up stale or you waste a ton of time and tokens keeping them up to date. And you can't even trust the LLMs to keep the docs up to date, you will still need to review it yourself keep a bunch of shit out of them.

kaydub··on Agents don't need memory, they need documentation
You guys show me a measurable difference on something with docs vs without docs and I'll believe it.

The docs just make you feel good. They're not worth anything. They just create more work if anything because now you don't only have to fight entropy in your codebase but also your docs.

kaydub··on Agents don't need memory, they need documentation
Don't agree. LLM generated docs are some of the worst because like I said, the LLM never trims, only amends. So we used to do X but now we've had an architectural change or some type of change where we should never do X. Instead of just removing the instructions to do X in the docs, it amends them, "we made the decision to no longer do X because of Y". Now it has X multiple times in context instead of just not having X in context at all.
kaydub··on Agents don't need memory, they need documentation
Honestly, don't agree. I think even this is a waste of time and context.

The LLMs will always have these suggestions on things that will "reduce friction in the future" but I really think the LLM is a sycophant glazing you. Because even when you add those skills or prompts at some point the LLM ignores it or does whatever it wants to do anyways. Better to leave most of that out and deal with it when it arises.

Even your example, instructions for running specs... none of the current frontier models need this at all.

kaydub··on The Illusion of AI Productivity: Go Frame the House
This is the pipe dream the LLM companies are selling.

But let's say you're right.

You're not going to join that "robot value chain" working at "ABC Framing company" and "ABC Framing company" won't be the place that comes out with the game changing piece to the "robot value chain."

You join the "robot value chain" by getting a job at US Robotics and Mechanical Men.

kaydub··on The Illusion of AI Productivity: Go Frame the House
I'm growing more and more frustrated at engineers wasting time building AI tools. I'm seeing a lot of people building these tools but not actually making anything. Documentation, memory, skills/plugins... I just don't think a lot of these actually DO anything and most people's time would be better spent using the LLMs to build software for people. Not more software for LLMs.

Anybody else noticing this? Anybody else hacking away all the bespoke tools, complex markdown files, and other extra stuff and just relying on the frontier models (and maybe an mcp or two to get your LLM controlled access to specific resouces)?

kaydub··on Agents don't need memory, they need documentation
The markdown files, documentation, decision files, (3rd party)memory systems, etc. are all a waste of time and context.

Every single project at work that has documentation/ADRs/etc ends up stale as fuck and it pollutes context more than it helps. Even the agent generated docs. Hell, especially the agent generated docs. It's like the LLMs can't just DELETE something, they always amend. So now there's stale details in there about something we abandoned polluting context.

The code is the documentation, none of this stuff is needed. I'm seeing WAY too many devs/engineers go down the rabbit hole of trying to build these tools and systems but not actually getting meaningful work done.

I've been doing my best to remove and delete all the documentation. I usually keep a super high level AGENTS.md/CLAUDE.md file and that's it. The llm/agent can figure it out. You might argue I waste 5 minutes and a dollar or two for the agent to "figure it out" each time, but I'm pretty certain people are wasting FAR more time and money building these things out and maintaining them in each of their projects (or NOT maintaining them and they're actively BAD for their projects).

kaydub··on Agents don't need memory, they need documentation
Yeah, I think monorepos are pretty great. Sometimes not possible, but for those situations you just need to have a way for the agent to get access to other codebases (gitlab mcp being what I rely on at work).

I don't think you need more documentation or these memory systems. The code IS the documentation. Lots of this stuff is unnecessary. It's a bunch of people doing their special rain dances and then when it happens to rain they say "I did that"

← PreviousPage 2 of 10Next →