Take a complex, sophisticated codebase you wrote, that only you have deep understanding of, and throw Codex or Claude at it and ask it questions. How does the Foo work? What happens if the user tries to Frozzle a Bar? Give me a guided tour of how the Baz feature was implemented, top to bottom of the stack.
Personally, I was amazed at how accurate the results were, on a codebase where I knew the answers already. Try to pretend that's not a useful tool to use when you have to fix bugs or add features to a large codebase you aren't already familiar with.
how will I know the answers are accurate?
It's similar to how Google has been useful for the past few decades despite the internet being full of junk info. Except Google could only provide semi-reliable information about stuff that's public knowledge with lots of people talking about it, whereas Claude can provide semi-reliable information about your own code that nobody else has ever laid eyes on.
A few months ago I used this to re-acquaint myself with code I myself had written back in 2017-2018[1]. I wanted to add some new features, and I did NOT want AI to write them, because the existing codebase was my own sweat and tears and running it through the slop extruder would cheapen that original work. I wanted to write all the code myself with full understanding of what was going on. Now, it had been many years and the codebase did some pretty esoteric stuff. I was the only person in the world who ever knew what that code did, and I didn't know it like the back of my hand anymore. Particularly the stuff with runtime CIL generation for result set processing.
I did a lot of reading my own code and a lot of thinking about it to get that understanding back, but it was a very helpful tool to be able to bounce my questions off an agent during the process and then investigate its answers for accuracy. And greater than 90% of the time its answers turned out to be correct.
Honestly, this reads a bit like you have not tried it, because the answer is 'kind of clear' if very nuanced.
You get a sense pretty quickly for the 'boundaries' of the tech - and though it will make errors for sure - you can sense what kinds of errors it's inclined to make.
Why? Unless you foolishly choose to target something like Javascript, or Python, or the like where the training corpus was full of poor quality code then the LLMs stay pretty true to how you'd write the code yourself. LLMs are only as good as what they were trained on, but luckily there was already a strong movement towards "all codebases should be the same" before LLMs, and as long as you hone in on those ecosystems then LLMs trained on those ecosystems can work quite well at producing quality code without the need for heavy oversight.
It depends on how much 'rails' there are.
More rails, then there is a lot less to examine.
You might find it to be an interesting read, but the software will be fine if you don't.
" You already know that the quality will be on par with what you would have written yourself."
No - I totally don't agree.
I've almost never seen a situation where one does not have to read, other than for bug fixes.
In almost every case where I have 'gone under the good' to AI created system - it was bad.
The AI is terrible at grasping and coding concepts. It names things poorly, and over iterations, it will turn 'state' into a hairball.
Try asking it to do an DOM event adapter to filter and map events, like doing 'ime end even guaranted' - it will do write something and write tests that all pass, but there will be holes.
The logic will be inconsistent and won't follow an hard spec, and even getting it to write a hard, clean spec won't work.
Let me put it differently: it's not the 'code' that's the problem - it's the concepts that it is trying to represent, which it screws up.
Only where those 'concepts' are already clean, such as in a really good code-base that you only need to tweak - does it work well. And amazingly well.
But for doing anything from scratch - you have to read the code.
I don't think it's going to get better any time soon because I think there is a 'noisy channel' problem here, in that it can only get so good without proper specification, and specification is 'the developers job' that cannot be replaced, at best we can get tools to support us there.
Why? While you can technically build a DOM event system in any language, you wouldn't, so this sounds like you are talking about Javascript (or Typescript, but same thing)... Which we already know LLMs cannot handle because the training data was full of poor quality code and thus LLMs also produce poor quality code. So, yes, you have to watch Javascript like a hawk here because the LLMs will go to crazy town if you don't pay attention. But that was already acknowledged.
Perhaps you accidentally chose the wrong example?
Second - it is very wrong to imply 'the AI has bad training data for JS / TS' - that's generally not how it works.
Third - the main point is not the code, it's information distillation as I suggested. It has nothing to do with the programming language. Yes - it might plausibly work better in some than others, but that's not the crux of the issue.
As to to your question the 'DOM event filtering' is just a good example because there are ambiguities in the problem space.
The spec is odd, browsers have different stated behaviours, different actual behaviours - this creates difficult problem space surface. It doesn't matter what 'la
It is an example, but an example that doesn't seem to fit the discussion.
> it is very wrong to imply 'the AI has bad training data for JS / TS' - that's generally not how it works.
Where would it have found good training data? The popularity of JS/TS means that the vast majority of the code that was written for it prior to LLMs is pretty lousy. Worse, even among the codebases that could be considered good, there is very little consistency between them so there is real good way for an AI to converge on what a good codebase even looks like in that ecosystem.
Regardless, no matter what explanation you want to choose, it remains abundantly clear that LLMs show time and time again that they struggle with JS/TS where there is no such struggle in certain other ecosystems. If you want to leverage LLMs to their full potential you are simply not going to be able to choose JS/TS (among others).
> As to to your question the 'DOM event filtering' is just a good example because there are ambiguities in the problem space.
Perhaps the disconnect here is that you have forgotten, but the discussion was about code quality. What about DOM event filtering in particular would lead an LLM that otherwise produces quality code to suddenly not? I have seen nothing to suggest that, in ecosystems that LLMs are suited for, that code quality declines when the problem space becomes more difficult. Code quality and problem space difficulty are orthogonal.
Or perhaps I am misunderstanding you, but my read here is that you point out that when a problem isn't well constrained an LLM may provide a solution that doesn't meet your expectations. That is true, but code quality and meeting user expectations are not one and the same. In fact, in the startup realm you often hear things like "users don't care about your code quality, just release something!" to really drive home that they are in entirely different dimensions.
And lacking in self awareness to publicize that.
Obviously some people are using it recklessly, it's the same with everything.
Do you think it's more likely that 'the entire world is wrong and you are right' ... OR ... that you just personally have not figured out how to use the technology in way that makes sense and/or you have a bad sense of of what quality means in most contexts.
AI produces easily 'better than human' code where it has a lot of guardrails, and is probably 10x better than humans at hunting down and fixing bugs.
Software is 90% 'maintenance' , and a well maintained code base provide ample basis for the AI to understand what design and formatting to use for a given solution.
It definitely falls down on it's own architecture, it's not that hard to work along with the AI.
It doesn't matter what we think of Altman or whoever, or whatever or AGI or 'the bubble' - that's all noise. It's literally the most blatantly obvious 'useful' thing to happen to tech since the advent of machines - it takes a 'collapse of intellectual contextualization' to not realize that.
You're watching cars replace horses, they are not the same thing.
Yeah, see, I've seen this movie before. I've seen it with Lisp. "You just haven't used it enough to reach enlightenment! We all have! Keep going; you'll see it too!"
And then with Haskell. Exact same looking down on everyone who doesn't agree as unenlightened, as people who Just Don't Get It.
Both Lisp and Haskell are useful languages. Neither is the future of mainstream programming, not just because the mainstream is unenlightened, but because those languages don't fit the way most programmers think.
People are different. Different languages fit different people. If Lisp fits how you think, when you find it after wandering through procedural languages, it feels incredibly freeing. But if it doesn't fit...
So I get very nervous about "you just haven't figured out how to use it" claims. Maybe it's true. Or maybe it just doesn't fit a bunch of people, and browbeating them with claims that they're unskillful does not make it fit them any better.
'Lisp' is an arcane tool and it's highly debatable in the objective sense if 'Lisp' is better.
The analogy is someone is saying that 'cars are useless' and that everyone should just keep riding horses.
I'm saying that we can debate the merits of cars, but it's absurd to suggest that they are not useful, and it's reasonable to conclude that someone who cannot 'find value' in cars, has not learned to drive them.
Then debate the merits.