Comment by simonw | Hacker News Reader

Comment by simonw | Hacker News Reader

Parent Full thread

> However, for more complex code questions particularly around less popular libraries which have fewer code examples scraped from Stack Overflow and GitHub, I am more cautious of the LLM’s outputs.

That's changed for me in the past couple of months. I've been using the ChatGPT interface to o3 and o4-mini for a bunch of code questions against more recent libraries and finding that they're surprisingly good at using their search tool to look up new details. Best version of that so far:

"This code needs to be upgraded to the new recommended JavaScript library from Google. Figure out what that is and then look up enough documentation to port this code to it."

This actually worked! https://simonwillison.net/2025/Apr/21/ai-assisted-search/#la...

The other trick I've been using a lot is pasting the documentation or even the entire codebase of a new library directly into a long context model as part of my prompt. This works great for any library under about 50,000 tokens total - more than that and you usually have to manually select the most relevant pieces, though Gemini 2.5 Pro can crunch through hundreds of thousands of tokens pretty well with getting distracted.

Here's an example of that from yesterday: https://simonwillison.net/2025/May/5/llm-video-frames/#how-i...

I think they might have made a change to Cursor recently as well. A few times I've caught it using an old API of popular libraries that have updates. Shout out to all the library developers that are logging deprecations and known incorrect usages, that has been a huge win with LLMs. In most cases I can paste the deprecation warning back into the agent and it will say "Oh, looks like that API changed in vX.Y.Z, we should be doing <other thing>, let me fix that ..."

So it is capable of integrating new API usage, it just isn't a part of the default "memory" of the LLM. Given how quickly JS libraries tend to change (even on the API side) that isn't ideal. And given that the typical JS server project has dozens of libs, including the most recent documentation for each is not really feasible. So for now, I am just looking out for runtime deprecation errors.

But I give the LLM some slack here, because even if I was programming myself using an library I've used in the past, I'm likely to make the same mistake.

You can just use @Docs [0] to import the correct documentation for your libraries.

[0] https://docs.cursor.com/context/@-symbols/@-docs

Sure, however I made the point "how quickly JS libraries tend to change" and "the typical JS server project has dozens of libs". That can add up to a lot of context.

I suppose a point could be made that I could keep the docs up-to-date using Cursor's custom docs feature and hope that whatever magic they are doing to manage context doesn't blow out the daily token budget. And I haven't tried that since the flow I mentioned about tends to work (as long as the libraries in question provide good diagnostic logs).

A good augmentation to my workflow might be periodically prompting "In file X I am using lib Y at version vN.M.P, check the docs for that version and report any errors or misuses ...".

I’ve had similar experiences in recent weeks, enabling search in ChatGPT with o4-mini-high to fix previously insurmountable hurdles I’d run into wrt breaking changes in libraries that have occurred outside the model cutoff dates, and for which the errors and fixes are non-obvious. It’s worked far better than I’d expected for what feels like such a simple UI toggle.

Reply on news.ycombinator.com