Hacker News Reader

Comment by GardenLetter27 | Hacker News Reader

I'm really skeptical of using current LLMs for judging codebases like this. Just today I got Gemini to solve a tricky bug, but it only worked after providing it more debug output after solving some of it myself.

The first time I tried without the deeper output, it "solved" it by writing a load of code that failed in loads of other ways, and ended up not even being related to the actual issue.

Like you can be certain it'll give you some nice looking metrics and measurements - but how do you know if they're accurate?

View on HN

> I'm really skeptical of using current LLMs for judging codebases like this.

I'm not necessarily convinced that the current generation of LLMs are overly amazing at this, but they definitely are very good at measuring inefficiency of tooling and problematic APIs. That's not all the issues, but it can at least be useful to evaluate some classes of problems.

I have mixed results but one of the more disturbing things I've found Claude doing is that when confronted with a failing test case, and not being able to solve a tricky problem.. just writing a kludge into the code that identifies that here's a test running, and makes it pass. But only for that case. Basically, totally cheating.

You have to be super careful and review everything because if you don't you can find your code littered with this strange mix of seeming brilliance which makes you complacent... and total Junior SWE behaviour or just outright negligence.

That, or recently, it's just started declaring victory and claiming to have fixed things, even when the test continues to fail. Totally trying to gaslight me.

I swear I wasn't seeing this kind of thing two weeks ago, which makes me wonder if Anthropic has been turning some dials...

I also feel like I’ve seen a lot more of these over the past week or two, whereas I don’t remember noticing it at all before then.

It feels like it’s become grabbier and less able to stay in its lane: ask for a narrow thing, and next thing you know it’s running hog wild across the codebase shoehorning in half-cocked major architectural changes you never asked for. [Ed.: wow, how’s that for mixing metaphors?]

Then it smugly announces success, even when it runs the tests and sees them fail. “Let me test our fix” / [tests fail] / [accurately summarizes the way the tests are failing] / “Great! The change is working now!”

Yes, or I've seen lately "a few unrelated tests are failing [actually same test as before] but the core problem is solved."

After leaving a trail of mess all over.

Wat?

Someone is changing some weights and measures over at Anthropic and it's not appreciated.

It is possible that the tasks you gave to the model previously were just about easy enough for it to handle, while the few failing tasks you gave recently were a bit too tough for the model, thus it had to cheat.

For the exact same task, some changes in the system prompt used by Claude Code, and/or how it constructs the user prompt, can quite easily make the task either easy enough or not. It is a fine line.

> identifies that here's a test running, and makes it pass. But only for that case

My team refers to this as a "VW Bugfix".

What do you mean that it "ended up not even being related to the actual issue"? If you give it a failing test suite to turn green and it does, then either its solution is indeed related to the issue, or your tests are incomplete; so you improve the tests and try again, right? Or am I missing something?

I explain this in sibling-node comment but I've caught Claude multiple times in the last week just inserting special-case kludges to make things "pass", without actually successfully fixing the underlying problem that the test was checking for.

Just outright "if test-is-running { return success; }" level stuff.

Not kidding. 3 or 4 times in the past week.

Thinking of cancelling my subscription, but I also find it kind of... entertaining?

I just realised that this is probably a side-effect of a faulty training regime. I’ve heard several industry heads say that programming is “easy” to generate synthetic data for and is also amenable to training methods that teach the AI to increase the pass rate of unit tests.

So… it did.

It made the tests pass.

“Job done boss!”

I found that working with an AI is most productive when I do so in an Adversarial TDD state of mind. As described in this classic qntm post [0] following the VW emissions scandal, which concludes with:

> Honestly? I blame the testing regime here, for trusting the engine manufacturers too much. It was foolish to ever think that the manufacturers were on anybody's side but their own.

> It sucks to be writing tests for people who aren't on your side, but in this case there's nothing which can change that.

> Lesson learned. Now it's time to harden those tests up.

[0] https://qntm.org/emissions

It made the other tests fail, I wasn't using it in agent mode, just trying to debug the issue.

The issue is that it can happily go down the completely wrong path and report exactly the same as though it's solved the problem.

Reply on news.ycombinator.com