sdesol

> will we still need vector-based retrieval

I think for most use cases, it doesn't make much sense to use vector DBs. When I started to design my AI Search feature, I researched chunking a lot and the general consensus was, you can can lose context if you don't chunk in the right way and there wasn't really a right way to chunk. This was why I decided to take the approach that I am using today, which I talk about in another comment.

With input cost for very good models ($0.30/1M) for Gemini 2.5 Flash (bulk rates would be $0.15/1M), feeding the llm thousands of documents to generate summaries would probably cost 5 dollars or less if using bulk rate pricing. With input cost and with most SOTA LLMs being able to handle 50k tokens in context window with no apparent lost in reasoning, I really don't see the reason for vector DBs anymore, especially if it means potentially less accurate results.

> An interesting alternative I've been meaning to try out is inverting this flow.

This is what I am doing with my AI Search Assistant feature, which I discuss in more detail via the link below:

https://github.com/gitsense/chat/blob/main/packages/chat/wid...

By default, I provide what I call a "Tiny Overview Analyzer". You can read the prompt for the Analyzer with the link below:

https://github.com/gitsense/chat/blob/main/packages/chat/wid...

In a nutshell, it generates a very short summary of every document along with keywords. The basic idea is to use BM25 ranking to identify the most relevant documents for the AI to review. For example, my use case is to understand how Aider, Claude Code, etc., store their conversations so that I can make them readable in my chat app. To answer this, I would ask 'How does Aider store conversations?' and the LLM would construct a deterministic keyword search using terms that would most likely identify how conversations are stored.

Once I have the list of files, the LLM is asked again to review the summaries of all matches and suggest which documents should be loaded in full for further review. I've found this approach to be inconsistent, however. What I've found to work much better is just loading the "Tiny Overview" summaries into context and chatting with the LLM. For example, I would ask the same question: "Which files do you think can tell me how Aider stores conversations? Identify up to 20 files and create a context bundle for them so I can load them into context." For a thousand files, you can easily fit three-sentence summaries for each of them without overwhelming the LLM. Once I have my answer, I just need a few clicks to load the files into context, and then the LLM will have full access to the file content and can better answer my question.

> Not sure I believe this. If you suddenly automate away 95% of any task, how could it be the case you retain 100% of your prior abilities?

I need to review like crazy now, so it is not like I am handing off my understanding of the problem. If anything, I learn new things from time to time, as the LLM will generate code in ways that I haven't thought of before.

The AI genie is out of the bottle now and I do believe in a year or two, companies are going to start asking for conversations along with the LLM generated code, which is how I guess you can determine if people are losing their skill. When my code is fully published, I will include conversations for every feature/bug fix that is introduced.

> The ability for a human being to communicate without assistance is extremely important

I agree with this, but once again, it isn't like I don't have to review everything. When LLMs get much better, I think my writing skills may decline, but as it currently stands, I do find myself having to revised what the LLM writes to make it sound more natural.

Everything is speculation at this point, but I am sure I will lose some skills but I also think will gain new ones by being exposed to something that I haven't thought of before.

I wrote my chat app because I needed a more comfortable way to read and write *long* messages. For the foreseeable future, I don't see my writing proficiency to decrease in any significant manner. I can see myself being slower to write in the future though, as I find myself being very comfortable speaking to the LLM in a manner that I would not to a human. LLMs are extremely good at inferring context, so I do a lot lazy typing now to speed things up, which may turn into a bad habit.

Not even remotely since the 5% that I need to write is usually quite complex. I do think my writing proficiency will decrease though. However my debugging and problem solving skills should increase.

Having said all of that, I do believe AI will have a very negative affect on developers where the challenge is skill and not time. AI is implementing things that I can do if given enough time. I am literraly implementing things in months that would have taken me a year or more.

My AI search is nontrivial but it only took two months to write. I should also note the 5% that I needed to implement was the difference between throw away code and a usuable search engine.

> Do people really try to one-shot their AI tasks?

Yes. I almost always end with "Do not generate any code unless it can help in our discussions as this is the design stage" I would say, 95% of my code for https://github.com/gitsense/chat in the last 6 months were AI generated, and I would say 80% were one shots.

It is important to note that I can easily get into the 30+ messages of back and forth before any code is generated. For complex tasks, I will literally spend an hour or two (that can span days) chatting and thinking about a problem with the LLM and I do expect the LLM to one shot them.

> Therefore, it doesn’t affect my work at all.

But that isn't what the author is talking about. The issues is, your good code can be equal to slop that works. What the author says needs to happen is, you need to find a better way to stand out. I suspect for many businesses where software superiority is not a core requirement, slop that works will be treated the same as non-slop code.

> 120K a year developer, if it makes them even 20% more efficient (doubt that), you have a ceiling of $2000/mo

I don't think businesses sees it this way. They sort of want you to be 20% more efficient by being 20% better (with no added cost). I'm sure the math is, if their efficiency is increased by 20% then than means we can reduce head count by 20% or not hire new developers.

It's actually more than 6 months. 6 months was when I developed enough to start chatting with AI to be really productive. Moving forward once the licence is in place and the files become unminifed you can track exactly what ai generated.

No I mean trying to start in a few years. Basically if you feel ai is a fad and are trying to wait things out.

Sure it can become a hobby.

> being forced on us all anyway, which kind of sucks

Business is business, and if you can demonstrate that you are needed they will keep you, for the most part, but business also has politics.

> probably monitoring how much we use the "AI" and that could become a metric for job performance

I will bet on this and take it one step further. They (employer) are going to want to start tracking LLM conversations. If everybody is using AI, they (employer) will need differentiators to justify pay raises, promotions and so forth.

Age discrimination, saturated market, no longer a team fit (everybody is using AI and they have metrics to backup performance gains), etc.

> It's all pretty subjective I think.

This is very measurable, as you are not measuring against others, but yourself. The baseline is you, so it is very easy to determine if you become more productive or not. What you are saying is, you do not believe "you" can leverage AI to be more efficient than you currently are, which may well be true due to your domain and expertise.

It summarized the instructions required to install and setup. It (Gemini and Sonnet) did fail to mention that I need to setup a server and create a DNS entry for the sub domain.

A lot of the messages are revisions so it is not as tedious as it may seem. As for the "3x to 4x", this is my own experience. It is possible that I am an outlier, but 80% of the generated AI code that I have are one-shot. I spend an hour or two (usually spread over days thinking about the problem) to accomplish something that would have taken a week or more for me to do.

I'm going to start producing metrics regarding how much code is AI generated along with some complexity metrics.

I am obviously bias, but this definitely feels like a paradigm shift and if people do not fully learn to adapt to it, it might be too late. I am not sure if you have ever watched Gattaca, but this sort of feels like it...the astronaut part, that is.

The profession that I have known for decades is starting to feel very different, in the same way that while watching Gattaca, my perception of astronauts changed. It was strange, but plausible and that is what I see for the software industry. Those that can articulate the problem I believe will become more valuable than the silent genius.

> I wish people could be a bit more open about what they build.

I would say for the last 6 months, 95% of the code for my chat app (https://github.com/gitsense/chat) was AI generated (98% human architected). I believe what I created in the last 6 months was far from trivial. One of the features that AI helped a lot with, was the AI Search Assistant feature. You can learn more about it here https://github.com/gitsense/chat/blob/main/packages/chat/wid...

As a debugging partner, LLMs are invaluable. I could easily load all the backend search code into context and have it trace a query and create a context bundle with just the affected files. Once I had that, I would use my tool to filter the context to just those files and then chat with the LLM to figure out what went wrong or why the search was slow.

I very much agree with the author of the blog post about why LLMs can't really build software. AI is an industry game changer as it can truly 3x to 4x senior developers in my opinion. I should also note that I spend about $2 a day on LLM API calls (99% to Gemini 2.5 Flash) and I probably have to read 200+ LLM generated messages a day and reply back in great detail about 5 times a day (think of an email instead of chat message).

Note: The demo on that I have in the README hasn't been setup, as I am still in the process of finalizing things for release but the NPM install instructions should work.

The issue is frontend can be a loaded question, especially if you are dealing with legacy stuff, different frameworks, etc. You also can't tell what the frontend code does by looking at that single line.

Now imagine as part of your analyzer, you have the following instructions for the llm:

--- For all files in `src/frontend/` treat them as frontend code. For all files in 'src/' excluding `src/frontend` treat as backend. Create a metadata called `scope` which can be 'frontend', 'backend' or 'mix' where mix means the code can be used for both front and backend like utilities.

Now for each file, create a `keywords` metadata that includes up to 10 unique keywords that describes the core functionality for the file. ---

So with this you can say

- `!ask find all frontend files`

- `!ask find all mix use files`

- `!ask find all frontend files that does [this]`

and so forth.

The whole point of analyzers is to make it easy for the LLM to map your natural language query to a deterministic search.

If the code base is straightforward and follows a well known framework, asking for frontend or backend wouldn't even need an entry as you can just include in the instructions that I use framework X and the LLM would know what to consider.

I am not sure I follow what you are saying. What would the line be and how would it become deterministically searchable?

Honestly, grepping isn't a bad strategy if there is enough context to generate focused keywords/patterns to search. The "let Claude Code think for 10 minutes or more", makes a lot more sense now, as this brute force method can take some time.

An MCP is definitely on the roadmap. My objective is to become the context engine for LLMs so having a MCP is required. However, there will be things from a UX perspective that you'll lose out on if you just use the MCP.

I've been able to create a very advanced search engine for my chat app that is more than enterprise ready. I've spent a decade thinking about search, but in a different language. Like you, I needed to explain what I knew about writing a search engine in Java for the LLM, to write it in JavaScript using libraries I did not know and it got me 95% of the way there.

It is also incredibly important to note that the 5% that I needed to figure out was the difference between throw away code and something useful. You absolutely need domain knowledge but LLMs are more than enterprise ready in my opinion.

Here is some documentation on how my search solution is used in my app to show that it is not a hobby feature.

https://github.com/gitsense/chat/blob/main/packages/chat/wid...

> But they have to get better at understanding the repo by asking the right questions.

How I am tackling this problem is making it dead simple for users to create analyzers that are designed to enriched text data. You can read more about how it would be used in a search at https://github.com/gitsense/chat/blob/main/packages/chat/wid...

The basic idea is, users would construct analyzers with the help of LLMs to extract the proper metadata that can be semantically searched. So when the user does an AI Assisted search with my tool, I would load all the analyzers (description and schema) into the system prompt and the LLM can determine which analyzers can be used to answer the question.

A very simplistic analyzer would be to make it easy to identify backend and frontend code so you can just use the command `!ask find all frontend files` and the LLM will construct a deterministic search that knows to match for frontend files.

But documentation doesn't include styling or preferred pattern, which is why I think a lot people complain that the LLM will just produce garbage. Also documentation is not guaranteed to be correct or up to date. To be able to produce the best code based on what you are hoping for, I do think having the actual code is necessary unless styling/design patterns are not important, then yes documentation will be suffice, provided they are accurate and up to date.

> current state of overspending on context

The thing that is killing me when I hear about Claude Code and other agent tools is the amount of energy they must be using. People say they let the task run for an hour and I can't help but to think how much energy is being used and if Claude Code is being upfront with how much things will actually cost in the future.

Thanks for the info. I actually want to make it easy for people to review aider, plandex, claude code, etc. conversations so I will probably look at importing them.

My goal isn't to replace the other tools, but to make them work smarter and more efficiently. I also think we will in a year or two, start measuring performance based on how developers interact with LLMs (so management will want to see the conversations). Instead of looking at code generated, the question is going to be, if this person is let go, what is the impact based on how they are contributing via their conversations.

Looking at the code, it does have some sort of automatic discovery. I also don't know how scalable Claude Code is. I've spent over a decade thinking about code search, so I know what the limitations are for enterprise code.

One of the neat tricks that I've developed is, I would load all my backend code for my search component and then I would ask the LLM to trace a query and create a context bundle for only the files that are affected. Once the LLM has finished, I just need to do a few clicks to refine a 80,000 token size window down to about 20,000 tokens.

I would not be surprised if this is one of the tricks that it does as it is highly effective. Also, yes my tool is manual, but I treat conversations as durable asset so in the future, you should be able to say, last week I did this, load the same files and LLM will know what files to bring into context.

I'm not sure if I would say human reasoning is 'probabilistic' unless you are taking a very far step back and saying based on how the person lived, they have ingrained biases (weights) that dictates how they reason. I don't know if LLMs have a built in scepticism like humans do, that plays a significant role in reasoning.

Regardless if you believe LLMs are probabilistic or not, I think what we are both saying is context is king and what it (LLM) says is dictated by the context (either through training) or introduced by the user.

I haven't looked at Claud Code, so I don't know if they have analyzers or not that understands how to extract any type of data other than specific coding data that it is trained on. Based on the runtime for some tasks, I would not be surprised if it is going through all the files and asking "is this relevant"

My tool is mainly targeted at massive code bases and enterprise as I still believe the most efficient way to build accurate context is by domain experts.

Right now, I would say 95% of my code is AI generated (98% human architectured) and I am spending about $2 a day on LLM costs and the code generation part usually never runs more than 30 seconds for most tasks.

Yes there is a context window limit, but I've found for most frontier models, you can generate very effective code if the context window is under 75,000 tokens provided the context is consistent. You have to think of everything from a probability point of view and the more logical the context, the greater the chances of better code.

For example, if the frontend doesn't need to know the backend code (other than the interface) not including the backend code to solve a frontend one to solve a specific problem can reduce context size and improve the chances of expected output. You just need to ensure you include the necessary interface documenation.

As for the full fledged app, I think you raised a good point and I should add a 'No lock in' section for why to use it. The app has a message tool that lets you pick and choose what messages to copy. Once you've copied the context (including any conversation messages that can help the LLM), you can use the context where ever you want.

My strategy with the app is to be the first place you goto to start a conversation before you even generate code, so my focus is helping you construct contexts (the smaller the better) to feed into LLMs.

LLMs (current implementation) are probabilistic so it really needs the actual code to predict the most likely next tokens. Now loading the whole code base can be a problem in itself, since other files may negatively affect the next token.