Local Indexing to reduce tool calling and token usage | Hacker News Reader

Local Indexing to reduce tool calling and token usage

tedhaley.ca

127 pointsTedHaley52 comments

I feel a lot of these tools are barking up the wrong tree. Solving problems that are just symptoms of a deeper root cause. Index because my codebase is too big to grep. Index to paraphrase hard_decline via card retries. Reduce tool calling to save my agent YOLO search loops. etc. etc.

It feels like trying to overcome bad codebase hygiene by throwing more tools at it, and wallpapering the mold over. I've found that codebases left entirely to LLMs suffer from this gravely, but with even a bit of effort - you'll get outsized returns. Don't let the agent smosh up everything into a giant ball of code. Separate concerns, build services, do the architecture step of the software design with a bit more care than vibe coding.

A bit of a rant, yeah, but I don't see why tools like this exist (and ask you to curl a script into bash, no less). You're doing stuff a bit wrong if grepping is the bottleneck.

I mostly feel the same, but it'd be really interesting to measure it, I think. Like, how many tokens did it cost in a codebase of quality Q1 vs one of quality Q2 to `grep` vs some semantic search. And did you find the correct `DuplicateSymbolName` in the correct namespace?

I don't think most people have control of the code base they're required to work in.

I too have converted a code base I made to typescript and peppered it with AI docs to make visibility easier.

I've converted i18n files to a k:v[] arrays per language because the AI is shit at working with objects. The ability now for it to just find/insert additional languages is well improved.

There definitely are shapes in code that the AI will have trouble with.

But again, I don't think most people have control over the code they maintain.

"Indexing" is a very general term and I don't think it is barking up the wrong tree.

A library card catalog system is an index. It exists to make it easy to find the right book quickly.

Your grocery store aisles are indexes to make it easier for you to find baking supplies from laundry and cleaning.

Well-organized codebases are already indexes by project, by folder structure, by naming convention, etc.

We use indexes everywhere, big or small.

The problem is really that the harnesses/SDKs/APIs themselves appear as if they weren't designed with indexes in mind from the outset.

You don't index because a code base is too big to grep. Grep can cut through almost any code base. The problem is that grep is lexical. It is dumb.

grep is insufficient or we wouldn't have LSP. At minimum you want some kind of structure aware search (for human ergonomics alone, but robots too).

>I feel a lot of these tools are barking up the wrong tree.

Its the right tree, but not the right branch. RAG systems are the next big thing. You can take a small model running locally, feed it a bunch of context and it will be able to generate the correct answer on par with a larger model with no context.

Context doesn't have to be documents/code either, it can be LLM generated context that guides next iterations of inference on how to do things.

OK, so, I too have built a local index tool called slopdex [1].

It's very simple, parses the repo code and docs with treesitter, indexes it, vectorizes everything, then allows you do things like create a map of the codebase or do a semantic search.

One cool trick that does work well is the cross-search command, which allows you to do a duplicate code search using the vector index. This gets rid of a blind spot that lots of LLMs have that make it create lots of dupliclate helper functions. This works quite well in practice.

It has a few evals that I'm still working on, showing only modest improvements in token usage when using the map command, maybe around ~10% less. I think this can depend a lot on the type of task though.

Would be happy to hear any feedback.

[1]: https://github.com/ninjaxtools/slopdex

I am interested to know your thoughts about publishing with the repository a natural-language implementation-agnostic specification that would allow other coding agents to create and mutate their own parallel implementations.

That's an interesting idea, though I believe that this should work just as well by pointing an agent at the code directly. At least, I don't see why a natural language sepecification would be better for this purpose.

Curious about this implementation. How does the vector search work with a versioned code base?

If I'm on commit A, then save the work and now I'm on commit B, how does cross-search ensure it doesn't return results for code JUST in commit A (i.e. code that is no longer in the codebase).

The index state corresponds to a commit hash and changing it (moving to commit B) will use git to compare the changed files and remove deleted files/functions from the index (and parse new ones).

Vector embeddings are cached, but added/removed from the index to correspond to the commit.

Are you modeling the actual API costs? If you can cut round trips at the end of a long context window you can end up reducing usage even if the total token count is the same.

That's a great point, thanks! I will include an eval for this case.

Nice tool. Does it support local embedding providers such as qdrant https://qdrant.tech/ ? Why only support paid, cloud-based embeddings provider ?

ChatGPT says it could work without updating the code, but since README doesn't advertise it, I was curious as to the reason behind this choice.

It says: "Can Slopdex run without paid embedding APIs?

Potentially yes, without modifying its code. [...] Slopdex supports a custom embeddingBaseUrl, allowing it to call a locally hosted, OpenAI-compatible embedding server. For example, with a local model producing 768-dimensional vectors: { "provider": "openai", "model": "your-local-embedding-model", "dimensions": 768, "symbolDimensions": 768, "embeddingBaseUrl": "http://localhost:1234/v1" }

Its code still requires an OPENAI_API_KEY value, but a dummy key should work if your local server doesn't enforce authentication.

The server must implement /v1/embeddings and accept Slopdex's request format, including the dimensions parameter. The relevant code is in vector_openai.rs."

Thanks, that's a good point.

I've not even considered adding this because I expect my underpowered laptop won't do well with local models, but it makes sense to add it.

https://github.com/huggingface/funes/ has been my go-to here (specifically for conversation search), but I do appreciate that there's a frontend on this one that can recall a session and open the correct agent for it (rather than my opening one agent, then opening another when I've gotten it wrong). Looks nice. Well done.

The Borges reference is neat. I’m a little lost in the sloppy documentation (“funes makes four deliberate, load-bearing choices”). As every project that Claude documents, figuring out what’s actually important is hard. I want to trust this one a little more than the average vibe code because it’s from HF.

My scepticism is that agent memory must be a non trivial problem to solve, or the labs would ship a solution in the harness. The models are happy to check against previous conversations if prompted, but they rarely do it on their own by default.

One advantage of plain text “diaries” is that you can read them as well, and the agent can add commit hashes, conversation IDs, links to experiment runs and so on. It works between models without any external tool.

Is "instructing users to ask their agent to curl a script and pipe to bash" the new "instructing users to curl a script and pipe to bash"?

If not for the rest of the page I would have assumed this was a joke.

I have seen this on at least half a dozen vibe-coded tool sites in the last few weeks

Goodbye, cruel world!

I saw like 20 tools of this kind and there is a new one coming every month. Cursor had something like this at some point. They've removed it

Referencing other sessions is a cool idea tho

> Cursor had something like this at some point. They've removed it

Any insight into why would be much appreciated because those insights might hold the key to whether this is a usable, sustainable idea to begin with.

I think a lot of people claim it is because in practice letting the AI just use grep yielded better results.

Hah yes. Ill throw my own into the ring too https://github.com/boyter/cs although its a lot older than most other tools having been started 7 years ago.

Cursor was an interesting case where they were maintaining an index. I think everyone has converged on not doing that, which is where I started, with BM25 with some tweaks on top.

Notable link related to this https://www.youtube.com/watch?v=ZnmtJKFonLk where Jo Kristian Bergum (notable in the search space) agrees with BM25 being a good approach for agentic workflows.

> Cursor had something like this at some point. They've removed it.

No, they say that they have removed it, but they haven't. Cursor claims they retired local indexing in favor of something they call "instant grep," which, according to their own support, doesn't use a local index, and they removed the tool in Cursor settings to rebuild the index.

But Cursor still uses a local index. They have just hidden it as a subfolder in your ".git" folder. "~/.git/cursor".

Their hidden index often goes stale, and I had to nuke that folder every time I changed many files. This isn't new; when they officially had a local index, it was often stale too. I developed a script that ran on a daily schedule, every few hours, to alert me when Cursor's index went stale.

I constantly had to watch for Cursor suddenly not finding documents that clearly existed, and it often worked on outdated content. Disabling instant grep removes the hidden index, btw and problem gone.

> Referencing other sessions is a cool idea tho

Something something if you're not already doing this you're behind.

I have a skill named read-opencode-db that's literally a 5 line md file as a simple documentation for a python script that allows agent to read other sessions

It's meant for me to reference session ID's directly to look at another agent review results for example

But I don't think that my workflow is well suited for allowing agents to freely grep the database for facts or decisions, I prefer Matt Pocock's domain modelling skill that bakes decisions into versioned and commitable markdown files

At some point if these types of front pages that are promoting a tool like this will be optimized for the agent itself to make the decision to install and use it. The human just says "please optimize for cost" and then the agent finds this thing and starts using it without the human even knowing or caring.

Humans fall prey to advertising and propaganda as well.

In general, in my experience, grep is better than almost anything else you can use. It's very very fast and efficient, very flexible, and does not require any indexing pass, is never out of date, and doesn't require you to guess ahead of time what files you might want to search or how to process search tokens.

"Search your files, repos and agent sessions, locally, and give your agents the best context for the job. One search points them to the few passages that matter, instead of grepping the whole disk."

Grep is already searching for relevant passages with a single tool call. This is clearly LLM written copy. It makes literally no sense.

This reminds me of the dating app advert I keep seeing on tiktok, which says that other dating sites "force you to be the loudest voice on your profile". There's no interpretation that makes sense, it's just the typical LLM gibberish.

Indexing local files makes search faster (in theory, but in practice maybe not for less than a few tens of GB of local files on fast SSD), but not more flexible or requiring less tool calls.

It also will always suffer from being a little behind the on disk state, which is terrible when you are trying to do something like find all the references to something under active development, or when changing branches. Consider you checkout a branch and call this tool, it will just lie to you, telling your LLM there are matches in files that no longer match, getting line numbers wrong, missing matches in other files. Every time you change branches it will go nuts trying to reindex everything. The best case scenario is still probably performance on par with grep but out of date and doing a ton of extra work to index files whether or not you ever need to search them.

Grep can take general regex, which more or less can't be indexed, so there's very few things that are more flexible or produce less tool calls than grep.

Grep is also absurdly fast, if you don't know how fast it is it's faster than you think. Here I use ripgrep to search every piece of code I have checked out, around 11GB of data (more or less) in 0.2s (regular gnu grep takes 4s, mostly because it doesn't exclude .git/ folders and only uses one processor):

  xyz@xx:~% du -sh checkout
  11G checkout
  xyz@xx:~% time rg robotic checkout|wc -l
  19
  rg robotic checkout  1.14s user 1.07s system 1151% cpu 0.192 total
  wc -l  0.00s user 0.00s system 1% cpu 0.192 total

Although I agree with all your points, I think there is a potential caveat with grep: it is not very token-efficient as it can return lots of matches, filling up the agent's context.

A potential solution for these indexing systems is to maintain several indexes (one per branch for example). Idk how they handle code changes though.

But I think the bigger negative signal is that these tools have been around for roughly as long as LLMs have, yet they are still barely used. Maybe because they are not that useful ?

LLM generated websites are so instantly recognisable

Vibe coded tools are great. I have loads of them to work for my workflows. AI makes it almost free to make your own tools. Almost none of them are worth sharing or distributing

And yet, if I present my boss with something that doesn't look like that, I'm not doing my job correctly. So, are LLMs sites bad or it's the incentive system just optimizing for less annoyances?

Insert picture of Lockheed Venture with red dots on it

I think there's a time-spent curve. If you put enough time into iterating and refining with attention to arch and design (weeks or months) then it might be worth sharing. If you've done it in days or a week, probably not.

How fast is yours searching for files? Is it support fuzzy search?

Does Pensieve reduce token usage by giving more relevant context to the agents, instead of agents searching and looking through lot of files ?

So basically this is ripgrep.

But someone will come in crying saying “Nooo!! It does so much more like indexing and bla bla etc”

Does this download an embedding model? If so which one? There is barely any information on how it works.

Where have API costs or agent token use hurt you most?

I don't know if you've noticed, but when Claude Code, Codex, or any other agent needs to find code, it runs grep. It searches your files for an exact string, reads a few results, and greps again until it finds what it wants. You pay tokens and wait for every round.

That worked fine when repos were small. But codebases are growing fast, partly because agents now write so much of the code. Every grep round costs more than it did last month, whether you notice it or not.

There's a second problem: grep only finds what you already know how to name. Ask it for "where we handle rate limits from the billing provider" and it gets 0% right on the first try, because it needs the exact string.

So I built Pensieve. It's a local index of your code, documents, and past agent sessions. Any agent can query it over MCP and get the few files that matter on the first try. A few cents saved per lookup doesn't sound like much. It adds up over hundreds of lookups a day.

Check out Pensieve!

(vouched - your comment was dead fyi)

I believe HN now has some built-in detection for AI generated comments, which is probably why it was dead (pangram rates it as 100% AI generated, btw).

I also built something similar to solve this, but more aimed at inspecting the impact of changes to a concept - think a vertical slice or domain, or even a feature. Often the same thing might be referred to by similar, but slightly different names, or separate pieces of the codebase might not immediately look related when they do share dependencies. So symbols, phrases, and words are attached to concepts for either LLM-based or user-based querying via MCP.

An indexer crawls the codebase and maps slices of code to concepts, then groups related concepts. You can inspect a concept to see things related to it, or ask about the impact of a change and see second-order changes required - things not so easy to see that are related by basic symbol linking (although it does use symbol linking - it heavily relies on Roslyn, since it's all C# focused).

It also returns confidence scores with everything, which adjust based on a dozen or so factors like whether the user defined the concept or the concept was auto-generated during the code crawl. Low confidence score === fall back to grep.

It was a super fun project and ended up being very useful, as much for people as the LLM. But, I'm not convinced it is materially better than grep in some ways and just haven't had the time to really focus on improving it.

interesting, I've been working on something similar myself... do you want to have a chat about it?

Sure! Send me an email, my name on gmail.

> So I built Pensieve. It's a local index of your code, documents, and past agent sessions. Any agent can query it over MCP and get the few files that matter on the first try

What are the various ways Pensieve is superior to iterative grep? Is it just better "semantic" search instead of literal "grep"?

Cache invalidation is hard - how does Pensieve address it?

Your site says "Pensieve watches your folders, repos and agent sessions and re-indexes what changed. Uncommitted edits count, and every worktree is tracked by branch, so your agents never work from stale context." but you don't define what "stale context" is - if I'm exploring 3 different implmentations of the same concept across 3 different branches, which one is the "stale context"?

Could you take a few minutes to explain the what and the how? Links to writeup to blog posts or files in your repo would be very helpful.

Finally, can I use Pensieve with Paseo? What if I wanted to expose Pensieve's capability to my Paseo setup?

This is 100% AI generated.

Why is it not an extension of Siri search, that is already available on all Macs, has a local LLM too?

this has made my M4 Macbook Pro with 24 GB RAM unusable.

Pensieve is just spamming it with new python processes. When it's not eating up my CPU cycles, it keeps crashing my machine. Did you test this with larger code bases?

Pensieve isn't ready for prime time yet.

tl,dr: Semantic search based on "Qwen3-Embedding-0.6B and stored in SQLite (with a trigram full-text index for exact matches) plus turbovec vector indexes".

Reply on news.ycombinator.com