I've got a lot of hallucinations like that from LLMs. I really don't get how so many people can get LLMs to code most of their tasks without those issues permanently popping up.
Edit: most LLMs are great for spitting out some code that fulfills 90% of what you asked for. That's sometimes all you need. But we all know that the last 10% usually take the same amount of effort as the first 90%.
Simple repetitive shit is easy to reason about, debug and onboard people on.
Naturally it's balancing act, and modern/popular frameworks are where most people landed, there's been a lot of iteration in this space for decades now.
After a few years those copy and pasted code pieces completely drift apart and create a lot of similar but different issues, that need to be addressed one by one.
My approach for designing abstractions is always to make them composable (not this enterprise java inheritance chaos). To allow escaping them when needed.
I mean is that bad? Unless you keep having to have huge MRs that modify every copy/paste could you just let the code sit there and run forever?
I only say this because I've been a maintenance programmer and I could only dream of a codebase like this. The idea that I get a Rollbar with a stack trace and the entirety of what the code actually does is laid bare right at the site of the error in a single file is amazing. And I can change it without affecting anything else?! I end up having to "unwind" all of the abstractions anyway because the nature of the job means I'm not intimately familiar with the codebase and don't just know where the real work happens.
The landscape (browser capabilities, backend stacks) has settled over last ten years.
We even had time to standardize on things somewhat.
Building new abstractions at this point is almost always the wrong move. This is one of the points that LLMs will improve in software dev - they will kill the framework churn because they will work best on stuff already in training data.
The issue is if you have an LLm write for you 10k lines of code, where 100 lines are bugged. Now you need to debug the code you did not write and find the bugged code, you will waste similar amount of time. The issue is if you do not catch the bugs in time, you think you gain some hours but you will get upset customers because things went wrong because the code is weird.
From my experience you need to work withan LLM and have the code done function by function, with your input and you checking it and calling bullshit when it does stupid things.
In other words, they're OK in use-cases that programmers need to eliminate, because it means there's high demand for a reusable library, some new syntax sugar, or an improved API.
As I've matured as a developer, I've appreciated certain types of boilerplate more and more because it's code that shows up in your git diffs. You don't need to chase down the code in some version of some library to see how something works.
Of course, not all boilerplate is created equally.
Edit: I'm not going to code asm just to be cool.
Every 'modern' project takes bucket loads of (annoying) setup and plumbing. Even in rather trivial 'start' cases, the LLM has to spend quite a bit of time to get a hello world thingy working (almost all of that is because 'modern programmers' have some kind of brain damage concerning backward compatibility; move fast and break things between MINOR versions that have no use or reason for those changes whatsoever but 'they liked it better', so stuff never works as it says on the product page; heaven help you if you need something exotic to be added). It's a terrible timeline for programming, but LLMs do fix at least that annoyance by just changing configs/slabs of code it until it works.
That sort of stuff can be very helpful to newbies in the DIY electronics world for example.
But for anything involving my $dayjob it's been fairly useless beyond writing unit test outlines.
You also have to be very explanatory with a direct communication style.
Our system imports the codebase so it can search and navigate plus we feed lsp errors directly to the LLM as development is happening.
Yeah I guess that would help a lot. Stuck with a bit more primitive tools here, so that doesn't help.
stuff like that works amazing
> But for anything involving my $dayjob it's been fairly useless beyond writing unit test outlines
This was my opinion 3-6 months ago. But I think a lot of tools matured enough to already provide a lot of value for complex tasks. The difficult part is to learn when and how to use AI.
Yep, if it's newbie stuff there are enough tutorials out there for the LLM to have data.
It's when you get off the tutorials and do the actual functionality of your project that they fail.
They can't, they usually just don't understand the code enough to notice the issues immediately.
The perceived quality of LLM answers is inversely proportional to the user's understanding of the topic they're asking about.
When I'm using llama.vim, like 40% of what it writes in a 4-5 line completion is exactly what I'd write. 20-30% is stuff that I wouldn't judge coming from someone else, so I usually accept it. And 30-40% is garbage... but I just write a comment or a couple of lines, instead, and then reroll the dice.
It's like working through a junior engineer, except the junior engineer types a new solution instantly. I can get down to alternating between mashing tab and writing tricky lines.
Prompt based stuff, like "extract the filtering part from all API endpoints in folder abc/xyz. Find a suitable abstraction and put this function into filter-utils.codefile"
My comment was more about just asking questions on how to do things you're totally clueless about, in the form of "how do I implement X using Y?" for example. I've found that, as a general rule, if I can't find the answer to that question myself in a minute or two of googling, LLMs can't answer it either the majority of the time. This would be fine if they said "I don't know how to do that" or "I don't believe that's possible" but no, they will confidently make up code that doesn't work using interfaces that don't exist, which usually ends up wasting my time.
But if I'm writing out a bunch of linear algebra, I get a lot of useful completions and avoid tediousness.
I've settled on Qwen2.5.1-Coder-7B-Instruct-Q6_K_L.gguf-- so not even a very big model.
A few months ago I tried to do a small project with Langchain. I'm a professional software developer, but it was my first Python project. So I tried to use a lot of AI generated code.
I was really surprised that AI couldn't do much more than in the examples. Whenever I had some things to solve that were not supported with the Langchain abstractions it just started to hallucinate Langchain methods that didn't exist, instead of suggesting some code to actually solve it. I had to figure it out by myself, the glue code I had to hack together wasn't pretty, but it worked. And I learned not to use Langchain ever again :)
- "Solve" the issue assigned to me with a bunch of code that looks about right. Passes review and probably not covered by tests anyway.
- Once QA or customers notice it's not working, I can get credit for "solving" the bug as well.
- Repeat for 0 value delivered but infinite productivity points in my next performance review.
I predict that the variance in success in using LLM for coding (even agentic coding with multi-step rather than a simple line autosuggest or block autosuggest that many are familar with via CoPilot) has much more to do with:
1) is the language a super simple, hard to foot-gun yourself language, with one way to do things that is consistent
AND
2) do juniors and students tend to use the lang, and how much of the online content vis a vis StackOverflow as an example, is written by students or juniors or bootcamp folks writing incorrect code and posting it online.
What % of the online Golang is GH repo like Docker or K8s vs a student posting their buggy Gomoku implementation in StackOverflow?
The future of programming language design has AI-comprehensibility/AI-hallucination-avoidance as one of the key pillars. #1 above is a key aspect.
Really?
Logging in Go: A Comparison of the Top 9 Libraries
https://betterstack.com/community/guides/logging/best-golang...
I don't think that we yet have one language that is good at all that. And yes, I (sometimes) program in Go for a living.
Let us say there is 3k lines of RFCs, API, documentation of niche languages, examples), 2k lines of code generated by Claude (iteratively, starting small), then I do exceed the limit after a while. In that case I ask it to summarize everything in detail, start a new chat, use those 3k lines and the recent code, and continue ad infinitum.
Claude has done a good job refactoring, though I’ve had to tell it to give me a refactor plan upfront in case the conversation limit gets hit. Then in a new chat I tell it which parts of the plan it has already done.
But a larger context/conversation limit is definitely needed because it’s super easy to fill up.
So no context, and differences of the definition of "large".
Perhaps if you come from Java, then yeah.
shrugs
I worked on a Python project which I'd consider as medium sized, ie not small but not large, and it was around 25kLOC.
My $dayjob is a Delphi codebase with roughly 500kLOC, which I'd say is large but not huge.
Though if you wrote it in something like K[1], then yeah ok, I'd agree 3kLOC probably counts as large.
You can fit a hell of a lot of functionality in 3k statements. Really whether it's considered large or small necessarily must rely on the functionality it's intended to provide.
What is the LOC of cgit? Because I made a lot of changes to it, too, albeit privately. I uploaded most files as project files.
https://github.com/williamcotton/webdsl
It's a pipeline-based DSL for building web apps with SQL, Lua, jq and mustache templates.
I'd say it's like 90% Cursor Composer in Agent mode.
This is probably more like a mid-sized project, right?
Common coding tasks are going to be better represented in the training set and give better results.
You still "used" an LLM to write the code. And it still saved you time (though depending on the circumstances this can be debatable).
That's why all these people say they use LLMs to write lots of code. They aren't saying it did it 100% without any checking and fixing. They're just saying they used it.
You write tests in the same way as you would when checking your own work or delegating to anyone else?
At some point it's not really worth anymore creating the perfect prompt, just code it yourself. Also saves the time to carefully review the AI generated code.