Show HN: Graphicle, a Rust port of Graphify with memory for coding agents
github.com
github.com
For the evals I've run (albeit, not super scientific), these memory systems did worse than the raw harness given ripgrep.
It seems like a really interesting tool and maybe future models will know how to use these tools more effectively, but yeah I remain skeptical.
The more I think about it, I suspect that either the training data or the finetuning process (or both) will have to incorporate these tools to really get benefit from them.
so your experience matches my own limited experience of starting to run evals of my own harness, which is that these specialized tools can help weak models but on the frontier they just make it slower and more expensive
I've always found memory systems icky, because it's difficult to tell whether a memory is still valid. Also, discoverability is always a concern (when is a memory relevant). This being both a code-graph and a memory system (seemingly unrelated things), actually addresses this in a novel way.
We compared a pre-release build of Graphicle's memory with 7 other memory tools (Mem0, Hindsight and others) on three public question sets. We wrote down the test rules before scoring anything, and used one open model to read and judge the answers for every tool, so it's the same setup for all of them. Details and caveats are in docs/benchmarks.md
What we haven't done yet is the thing you're really asking: does a coding agent actually finish more tasks with these tools? We could use the same agent and model on the same tasks, with and without the tool, and compare how many tasks pass, tokens used, tool calls and time.
Curious what you're using today, and what result would convince you?
I have only just begun playing around with evals for a harness I'm working on and so far I have just been doing runs of converting https://github.com/realworld-apps/realworld from one language to another
not super pleased with it
the unsatisfying answer I think is that crafting good eval environments for long runs takes a lot of time, and there are many things you'd want to verify that you cannot simply lint or test for
Letting another LLM decide quality probably isn't the only meterstick you want, either.
Right now if you want your coding agent to understand your code and remember things, you end up stitching tools together. Graphify for the code graph. Mem0 or Hindsight for memory. Each one runs separately, stores its own data, and you wire them all into your agent yourself.
Graphicle puts both in one tool. It maps your code, remembers your docs, decisions and past agent sessions, and gives your agent the relevant bits of code with file and line. One install, runs locally, no account, open source.
The code map part is a Rust port of Graphify (thanks Safi and the Graphify folks). On Kubernetes, it built the code graph in about 3 min vs 26 min for Graphify, same result.
The memory side builds on open-source work from Mem0, Hindsight, Graphiti, MemOS and GraphRAG, all credited in NOTICE. Thanks to all of them.
Try it:
brew install graphicle-ai/tap/graphicle
graphicle setup
Would love to hear if setup worked in your repo and what's missing. I'll be here answering questions.