Comment by jonahx | Hacker News Reader

Comment by jonahx | Hacker News Reader

Parent Full thread

> have the LLM focus and summarize a slice of the system when they need to know about it or ask for it. But.... That's not knowledge? That's just search a la google? And thus seems to be subject to the same problems where the quality of your search dictates the quality of your knowledge that you surface at any given time?

This is the crux. Take pause, because hidden here is a sea-change in our fundamental trust model of software...

There is no longer anyone to turn to if this system breaks by, eg, having a critical bug that no one can figure out. No causal model in anyone's head that can be appealed to, to slowly and methodically debug. By design, the system is too large for that, and everything hinges on Claude itself being able to find and understand the right slice of the system in any situation. As long as that very new invariant holds, everything will be all right. If it fails catastrophically, there is no recovery. You are the owner of a recovered alien spacecraft with tech you have no hope of understanding.

The new trust model is: We trust Claude to always understand the system enough to debug anything or make any reasonable change we could want. The risk might be worth the reward. But it's worth explicitly laying out the deal we're making.

Tbf, it's worth pointing out that many large legacy systems already work like that (sort of), with "Claude" replaced by "whoever happens to be maintaining it this year".

"Tbf, it's worth pointing that many large legacy systems already work like that (sort of), with "Claude" replaced by "whoever happens to be maintaining it this year".

The real difference as I see it is that in the human maintained old style system, the human would take things slowly and ensure they understand any changes. Claude let loose will just say "yes, I know how to do this" and bork the system, because it doesn't have a concept of humbleness in the face of complexity!

Not only that, when you have a human designed and implemented system you know it's at least coherent enough that someone understood it at some point well enough to take responsibility for it. No such assurance with the llms.

But is this really a problem? If you have testing and you have version control via Git, you can roll back changes and investigate them.

I also don't even know if it's really true. Sure, maybe it's not coherent enough that a person understands it, but if an LLM wrote it, you can be ~99% certain (in my experience) that that LLM will understand it, and usually in less than the time it took to write it. And unlike humans, the LLMs don't die and take their knowledge with them, or change companies, or get ornery when you ask them to explain.

Also, in my experience, Claude doesn't really do what the parent is saying and dive head-first into complex problems and bork your code. It is fairly humble in the face of complexity. It constantly recommends breaking down larger problems into smaller ones that are more easy to grok.

But most importantly, anyone who's a decently skilled software engineer and who's using LLMs to write code is going to be thoughtful. Especially when they're writing code that's important. Unless you're an AI researcher pushing the bounds of what LLMs can do, which let's face it, the vast majority of programmers are not, it's almost always the case that you'll have your LLM writing intelligible code that you can understand and have it explain easily enough.

> can roll back changes and investigate them

Sure, you can do that, but irreparable harm might have been already caused. Sure, that’s unlikely when your product is an LLM wrapper. But in something like a database, it could be catastrophic.

I mean it could be, sure. But I think people are generally speaking more careful with higher stakes code than they are with lower stakes code. I know a lot of people who are YOLO'ing it with LLMs, but who ask for more understanding, more reviews, more tests, etc., in trickier areas. Which is only natural. Just like people are more careful when carrying a baby than when carrying a pillow.

(Yes, there will be individual exceptions to this, just like with everything else in life.)

> Just like people are more careful when carrying a baby than when carrying a pillow

lol, I assume you’re not a parent then. Or at best a new parent.

What do you mean, exactly?

They're talking about operational benefits of having backup kids.

And rolling back changes

> anyone who's a decently skilled software engineer and who's using LLMs to write code is going to be thoughtful.

Is this a definitional argument? As in, if you’re not thoughtful, you must not be a decently skilled engineer? Because I see no reason to believe that people are going to be thoughtful about LLM-generated code. Just the opposite. It’s incredibly tempting to LGTM it.

> I see no reason to believe that people are going to be thoughtful about LLM-generated code.

I think that's true for low value code. But the more important and crucial and high-stakes code is, the more effort people put into it, generally. Even with LLMs.

I would love to know what the Venn diagram of people who complain about people not understanding their code, to people who can't describe the boot process of their system is.

"their code" == code they actually write, that originates with them, "their system" == anything, from gentoo to something off-the-shelf

So the real question would be, how many people designing / implementing the boot process of any given system don't understand what what's going on.

If anthropic is making as many PRs as stated in articles, then rolling back is unlikely an option. More likely it'd be a new commit to fix it, and with a feature flag to preserve the "wrong" behavior (even if wrong, since people will probably depend on that function since its original behaviour).

Those sorts of changes are really rough to manage. Maybe still manageable by LLMs but we'll find out.

Is the Hugging Face incident from July really a problem? Or Australian healthcare portal breach? Who let their guard down unable to detect or roll back the change for days on end?

As I said at the end of my comment, unless you're an AI researcher pushing the bound of what LLMs can do. I don't think most of us are running ten million dollar AI swarms with unreleased models that are escaping their sandboxes. That's research, not software engineering, and this discussion is about whether or not software engineers can reasonably expect to understand their LLM generated code.

Also, for what it's worth, LLM research is hardly the first scientific research to have accidents with negative external consequences.

Frontier labs are not the first to socialize externalities and privatize the gains. Not even the first to shift the responsibility on to some non-entity (agents, corporations). I think you have a very reductionist view of software. Software is used to design the buildings you inhabit, bridges you cross, vehicles you drive, medical devices you wear...

I'd like that to be true, but many software service startups were stitching together node packages and selling the result long before LLMs became useful.

I am not sure that "someone understood it well enough to take responsibility" means what you think it means. I have definitely come across old systems which have had addons bolted on to them in so many places that the overall behavior of the system is unpredictable and only mostly works.

"I've seen bad code in the past" does not address the fact that human code by an experienced software developer is almost always less complex and more clear, with less repetition, dead code paths, and odd architectural choices than what an llm generates. If you don't recognize this, then you either haven't been reading the code or are not an experienced software developer. 80% success on a benchmark means they fail 20% of the time in the exact domain they were trained on.

I am an experienced software dev, and I am telling you this as clearly as I can: human generated code is oftentimes buggy, indecipherable slop.

You need to put 80% in context. There's a reason why FizzBuzz lasted as long as it did - lots of developers, even senior devs, are terrible at writing code, subjected to absurd time constraints, or required to color inside the lines of a leaky abstraction. Sometimes all three!

I've seen many situations at my current job where asking an LLM to do a refactoring made things much better - but if I had to do it by hand I'd just give up and move on to more essential work.

> you know it's at least coherent enough that someone understood it

30 years working in this industry and I'd say that's the exception rather than the rule. When it comes to large systems.

Yes, [citation needed]

I'm guessing you've only ever worked at startups?

Large company codebases are very much not like that. I've worked on systems (e.g., banking) that have accreted still-running code for as much as 60 years. Nobody understands how it works. Nobody ever understood how the whole worked, and many of the parts were never actually intended to work together.

And this holds true for most of the Internet. There is no one person who truly understands how it all works: DNS, routing, and all the other things we rely on to get our systems to work.

Network engineers and TCP/IP stack implementers have pretty good understanding of how the Internet works.

But I have see quite terrible finance software, patches over patches hold together by a duct tape. Also finance software has to implement large complexity of tax law, which is very large and always changing. It also doesn't help if the company developing finance software sees software development as a cost center and wants always to push the costs down (especially short term costs).

This is mostly a self made and kept problem. I also worked in codebases where the current developer(s) didn't know why things were the way they were and would take days to dig into why some part wasn't working - but the issue was, instead of taking the time to write tests and document, even if only when doing these point in time fixes, they just did the ad-hoc thing and solved whatever issue there was at the time without any worry whatsoever.

At some point I was tasked with improving the queries/retrieval times and started converting parts of the code and not only was it possible to write tests that would convey the actual meaning of the code and crystallise it but I started finding bugs in the implementation that no-one realized were even bugs because they no longer knew what it should do when queries grew to certain complexity and the search space was in the hundreds of millions of rows that thinking through them just with napkin calcs wasn't cutting it (specially when the bugs were subtle and the results still allowed the program to work "mostly" correctly).

Sure that happens, but over _years_. LLMs can make this happen over _days_.

Someone understanding something and the codebase being coherent are different things. I can only think that you have been lucky with your career placements if you are making that statement.

What you are saying is leaning on 2 premises that seem to not hold water 1. Code written by people is always more readable and maintainable than LLM code 2. Code written by an LLM is always less readable and maintainable

Can you really make those statements in all intellectual honesty?

False assumptions. The human doesn't have to always be more readable, to be usually more readable. And vice versa. LLMs given the opportunity to generate a "feature" almost always generates more complexity than a software engineer would develop.

Here is an explicit example of the additional lack of coherence due to LLMs in the Anthropic Claude Code public repo itself: the human maintainer trusts LLM's incorrect understanding of the current state of code in the repo.

The comment https://github.com/anthropics/claude-code/issues/30407#issue..., written by Claude Code, and posted by the human GitHub account says:

> "Issues here are auto-closed by a bot when they've had no activity for a while and are missing information needed to reproduce (usually a version and steps, or a /feedback ID). Nothing is triaged solely by counting"

In reality all issues in this repo are subject to time-based auto-closing, unless they collect a number of upvotes. See details at https://github.com/anthropics/claude-code/issues/87647.

Therefore, I wonder whether the internal review process described in the Anthropic's blog for internal repos also results in such issues, and whether Anthropic has a internal way to measure and mitigate that. The blog post says:

> "When someone wants to understand a part of the codebase, they ask Claude to write an explainer for exactly that slice. It takes minutes and reflects the code as it is today."

The Claude Code public repo problem shows that LLMs don't always "reflect the code as it is today", and can be a source of additional misunderstanding.

Exactly this. Using an LLM to understand the original LLM slop is just another chance to get it wrong 20% of the time. These things just aren't reliable.

> You are the owner of a recovered alien spacecraft with tech you have no hope of understanding.

I love this phrase, and I think it captures the difference between the "60 years of legacy code" mentioned elsewhere here, and the vibe-coded monstrosity that we all know we're going to be encountering in the next 5 years. The fact that there were human brains with intent making all the little (short-sighted, myopic, ignorant, etc) changes in the legacy code means that you DO have at least some shot of gaining enough context to make safer changes.

There is no real intent in LLMs and particularly in blind vibe coding, so you'll end up reading stuff that works, or appears to work, but there was no human brain behind the change for you to model and thus predict.

This just means we can't take the (often bad) shortcut of guessing what the code does because we feel we can predict what the author was thinking. Instead, we have to treat existing code as a natural sciences exercise - use the same methods we apply to the world around us, which has its logic and structure, but no human intent behind it.

GP is saying that "the search space" of "the natural world" is much bigger than the search space of "stuff created on purpose by humans". In both cases, there will be guesswork, but in one case less.

I think there is something to this, but am even more concerned by the pure scale. Consider 100k, 1M, and 10M LOC human legacy projects (already a big problem, usually!), and now compare to the likely outcome of 100M, 1B, 10B, etc LOC llm projects. That's where you may start to see the outcomes I reference in my OP.

> GP is saying that "the search space" of "the natural world" is much bigger than the search space of "stuff created on purpose by humans". In both cases, there will be guesswork, but in one case less.

I agree. What I am saying is, we have effective tools and processes for each of these search spaces, so code no longer being "created on purpose by humans" is not the end of the world.

> we have effective tools and processes for each of these search spaces

Do we? Science (the tool for the natural world) is limited and there's so much we don't understand.

The recent superhuman achievements by llms in math are promising, but what we've seen so far isn't enough imo to YOLO building arbitrarily large systems in code and just trust that the llms will be able to handle it. Especially when the stakes are high (human lives, etc).

> Do we? Science (the tool for the natural world) is limited and there's so much we don't understand.

Limited in what way? Its basic structure is sufficient to eventually let us discover anything there is to discover. It's a fully general problem solving framework.

> Limited in what way? Its basic structure is sufficient to eventually let us discover anything there is to discover. It's a fully general problem solving framework.

That "eventually" is doing a lot of heavy lifting. It's limited in the most literal, empirical way. We don't have answers to most things. If we stood in relation to a gargantuan code base as we stand in relation to the natural world, with an equal proportion of known vs unknown, we'd be pretty screwed.

As an aside, even with the "eventually" qualifier, the claim isn't true. The generation of hypotheses, in particular, and even sometimes the experimental setups to test them, happens through an "out of band" process -- namely, human creativity and intuition. Which is effectively a computational process we don't understand, and limited in its own way (despite its incredible productions). The history of scientific breakthroughs is the history of astonishing insights out of nowhere, from the brains of individual people, sometimes literally from dreams. The experimental method provides a reliable way to test the insights more or less "mechanically", but it would be mostly impotent without the other part.

most large software companies are already too large for anyone to have a complete mental model of.

The idea that someone at say, Google, will be able to fit even half of the system architecture in their heads is not realistic. It also hinges on being able to find and understand the right slice of the system in any situation.

But we have all sorts of guardrails to make that easy - i.e logs, traces, etc

Right, that's what I alluded to in my last sentence.

It is true that this gamble has existed in the form you mention for many years. Still, I sense something new here, both in scale and in the attitude of trust. Also, I find those massive, incomprehensible human-built systems frightening too.

Having worked continuously this decade on one massively complex consumer facing app that interacts heavily with real-world scenarios, I can say that the problems of cohesion existed well before the agentic revolution. Tons of features built by myopic teams simply because they couldn't understand enough of the system (both online and offline, meat-space interactions) to make good design decisions. AI actually helps here by being able to perform deeper reading and analysis than any human can do. But then the risk is that it can also produce way more random non-cohesive features.

So far I kind of feel like it nets out to the same bottleneck being good judgment from decision makers. And you might say, no, leadership will just put more and more pressure to add more complexity just because we can and it makes it appear that we are moving fast. But when it comes to real-world apps, there is a backstop which is that things that are too complex and illegible end up not being fit-for-purpose for human problems.

Claude Code and other general purpose AI tools are not like this, because they are literally trying to optimize it for every single kind of knowledge work with zero constraints on use case or target audience other than chasing the next juiciest one they can with essentially unlimited resource. 99% of software is not like this, and when the dust settles I suspect software won't be anywhere near as different as the doomers or the maxxers are predicting right now.

One problem is managing expectations from non-technical people. You can ship faster but you are also producing tech debt at that same faster rate, and possibly even faster as the gaps in your initial implementation/prompts are filled by Claude deciding to do random shit.

In most orgs, there is also a gap between leadership, even tech leadership and people actually using LLMs every day. And it is challenging for actual devs who are being told that they should be going much faster and the reality which is more mixed. Pushing back on people who are convinced that Claude is God is going to be tough if you are a junior.

I don't feel the job has really changed, I don't write code as much but that time goes into working out the structure...which is what I was doing anyway. But it has changed the general environment, how work is planned, expectations, etc.

I don't think they're the same. AI code isn't just complex because it's huge.. it's complex because it makes no sense

If you have a large human-written system with a little thought put into it you should be able to find the specific pieces that handle the behavior you want and work on that, and if it's good then changing it should have a small blast radius and be covered with regression tests

In contrast, I've fully vibe coded a 15k line tool for myself with Opus and I have no idea how on earth it's architected after reading through it. It has "tests" but I don't trust them. It's just all over the place with no coherent design to it. It works, but it's making me more and more nervous to change it

You can tell with some of the more recent models that they have been fine-tuned in such a way to produce a lot of descriptions that sound impressive but have no real meaning. In terms of production of semantic text, the recent models are significantly worse than a year ago.

It is not completely useless as a tool but there is still a huge gap in most real-world applications (I work in an org with multiple thousands of devs and I am top 5 AI user consistently, I have used Claude Code since pre-release). If you are building something like an SQL database which has decades of logic tests built by humans then it can work. If you do not have this (i.e. 99.99% of the time) then you still need to be able to issue prompts that give context. And if you start without that context, it will produce shit code in extremely basic scenarios. In particular, humans are far better at producing code that is easy to change (in practice, you need to create unintuitive, non-human rules to govern this i.e. creating many artificial boundaries which would be completely unnecessary for humans).

I am not sure whether the models just capped out in terms of coding or whether they are just trained by people who have never worked on enterprise software but the gap isn't closing. It has just been filled by people who lack the knowledge to identify when they are being bullshitted (the prevalence of consultants suggests that this markets is large).

I have worked on legacy stuff before. In most cases, there is a kernel of reason and logic that can be recovered. I have seen pure AI projects and they are the worst grade of TCS-authored abandonware. They can often do one specific thing but are basically unmaintainable outside of that.

> You can tell with some of the more recent models that they have been fine-tuned in such a way to produce a lot of descriptions that sound impressive but have no real meaning. In terms of production of semantic text, the recent models are significantly worse than a year ago.

I frequently find this in LLM generated comments, both when using an LLM myself or when reviewing code that someone else used an LLM for. I find it is often the most obvious indication that nobody really read through what was generated.

Yes, it has become quite common that Claude’s text feedback, e.g. Claude found an issue in the existing code that requires a decision and Claude tries to explain it, it is a paragraph of bullshit words stringed together.

Then you investigate it and it is often something relatively simple that could have been explained with short sentence.

Have you found a way to stop this?

I used that engineering language skill which is good when it uses it...but it doesn't like to use it.

...there are clearly some areas of alignment that they are not looking at.

We had one that hallucinated benchmarks, with graphs and everything! (The code didn't actually work.)

I'm trying to learn how to write software with agents where I keep control over the design while yielding control over the implementation.

This is not very different from what already happens to architects that delegate implementation to teams.

The challenge Is to find a way to encode the architecture in a way that both you and the agents can understand and agree on.

I'm still learning how to do this effectively, but my initial experience is quite positive. Not because agents are great but because humans are bad; the degree of coherence I've seen agents following is bigger than many human teams I've led in my career.

My benchmark is a quite big project for which the architecture is already well known, namely the reimplementation of the OpenVMS OS (kernel and user-land). There are many traps and many opportunities for the agents to just go ahead and diverge from the goals, so I found it a good playground toearn how reign them in but let them be effective at "drawing the rest of the owl"

I vibecoded a project without looking at the code as an experiment, and I had the same experience as you. It really drives home the point that, left to their own devices, LLMs produce abysmal code.

If you give them no constraints, they’ll produce code that just works, and it’ll look exactly like that.

We already know the solution to this problem. Software engineering. Put the LLM through the same grinder as any software dev and out constraints into place. Tell the LLM that the code design is bad and that it should improve it.

In my case, I had to go through several rounds of refactoring before the code at least resembled a good codebase.

A fun fact is that the refactorings didn’t break any functionality. Only adding new features broke preexisting behavior.

I don't think this is the case anymore if you're using the latest expensive models, i.e fable.

i still havent seen claude write code as bad as legacy spaghetti, but i guess im never quite vibe coding since i usually have an idea of the architecture i want, and that leaks into my prompts about how it should approach the problem and verification of the solution?

One of the fun things that you currently need to babysit Claude, etc, on, is that while they've been tuned to hell now to do more testing and even logging and such, they don't currently do enough log-and-debug-oriented specific passes/revisions to distinguish meaningful logging for meaningful debugging from noise that won't necessarily tell you what you need to know.

And you can ask them to address this. And you can ask them to look for that in all your PRs. Etc etc. And that helps a lot.

But it's not something that can be brute-forced in the same way "this failed a test, let me fix it" is. So without a long-term-memory brain holding all the pain of past incidents and knowing and applying ahead-of-time judgement about what to build, it's a challenge to turn the general guidance into the specifics for any unique piece of software.

They're trying to get there of course. We shall see what's next.

("Human code turns into this too" is also a curious rejoinder from the "don't read the code anymore" crowd because there are entire disciplines and areas of study that have emerged and evolved from the problems of large ad-hoc multi-faceted systems being difficult and risky to deal with.)

> There is no longer anyone to turn to if this system breaks by, eg, having a critical bug that no one can figure out. No causal model in anyone's head that can be appealed to, to slowly and methodically debug.

At this point, why even bother with source code? Source code is designed for humans to comprehend. If humans are no longer the intended audience, why not have machines output and maintain some more efficient intermediate representation of computer programs?

I fixed many bugs over the last years with help of claude code in an unfamiliar codebase. it is never just Claude alone so we don’t rely on AI understanding 100%. It is whatever AI can understand plus what I do. If you use AI right your understanding will also grow over time.

Reply on news.ycombinator.com