Comment by the_mitsuhiko | Hacker News Reader

Comment by the_mitsuhiko | Hacker News Reader

Parent Full thread

> I find this workflow makes my life extremely tedious, because reviewing and fixing whatever the machine produces is a slog for me

I don't actually mind the code review process, it's what I'm used to with open source contributions. Not infrequently did I look at a PR that people left me and decide to rewrite it entirely anyways because I did not quite appreciate the method they chose. That however did not mean the PR was useless, it might have provided the necessary input that made me go down that path ultimately.

I also think that what can make code review enjoyable is not necessarily because the person on the other side learns something from it. I actually tend to think that this something that does not necessarily happen in PR review, but in 1:1 conversations in person or though more synchronous collaboration.

So really my answer is that I do not mind code review. That said there is an important part here:

> How do you reconcile this with your own work? Maybe you just skim the output? Or do you run it and test that way?

What makes agentic workflows work is good tooling and fast iteration cyclces. That is also exactly what a human needs for a PR review to be actually thorough. I cannot accept a change when I do not know that it exhibits the intended behavior, and to validate that behavior I usually need a test case. Do I need to run the test case myself? Not necessarily because the agent / CI already does for me. However I do sometimes run it with verbose output to also see the intermediate outputs that might otherwise be hidden from me to give me more confidence.

It really is hard to describe, but when you put the necessary effort in to make an agentic workflow work, it's really enjoyable. But it does require some upfront investment for it to work.

> Please don't tell me you just trust the AI to write automated tests or something...

I do in fact trust the AI a lot to write tests. They have become incredibly good at taking certain requirements definitions and derive something that can be mostly trusted. It does not always work, but you get a pretty good feeling for if it will or not. At the end of the day many of the tasks we do are basic CRUD or data transformation tasks.

> It really is hard to describe, but when you put the necessary effort in to make an agentic workflow work, it's really enjoyable.

I wish it wasn't hard to describe because every attempt I've made to reach that point with agents/tools-in-a-loop has ended with more frustration, more errors, and more broken code than when I started, even with common and small, tedious tasks. I'd very much like to understand what proponents are doing and seeing that works.

It would be interesting to watch a video with an example of effective work. To see why are we so unlucky with results:)

All too often with AI proponents I get the you’re holding it wrong vibe. I can’t help but develop the suspicion these people are putting out bad work and just never notice. Or to be a little less scathing, fine, there’s a very narrow range of tasks where these models can usefully alleviate drudgery—quite often that comes at the cost of code bloat, since a generative model inherently makes it easy to puke out code with total indifference to its burden over time.

there was literally a post a few days ago where someone from cloudflare or fly (can't remember) released a new feature built with AI (albeit heavily supervised) and answered q's along the way about it.

it's funny because what i'd also like to see is people who are skeptics make a video as well since sometimes i also have the opposite suspicion. i get a lot of the criticism, but i don't get the "it produces pure garbage" type ones.

My comment was admittedly a little inflammatory, so I’ll try to elaborate on what I find so exasperating about the whole experience of using LLMs for coding (in any form, chat, agent, etc.): at root, it’s their total lack of a model of computation.

This leads to mistakes that are difficult to catch, because while I know the kinds of mistakes another human might make, having been through the process of learning not to make them myself, LLMs produce whole classes of bizarre mistakes that I have no interest in learning to catch. There is no discernible flawed mental model behind these errors—which with a human could be discussed and corrected—just an opaque stochastic process which I can tediously try to set on a better course with ‘incantations’, attempting to dial in to a better part of the training data that avoids the relevant class of error.

It’s honestly amusing how much of ‘prompt engineering’ (if it can be dignified with that term) amounts to a modern-day kind of mysticism. What better can we really do though when these models’ structure and operation is utterly opaque, on the one hand through deliberate, commercially-oriented obfuscation, and on the other because we still just cannot explain how a multi-billion parameter model works.

It’s rewarding to work with human juniors because people actually learn and improve. My learning how to coax these models into producing better than trash just is not, especially on anything either remotely novel, or in a legacy codebase that requires genuine understanding of how an existing system functions at runtime.

Once those two ends of the spectrum are ruled out, I find there’s little left that an LLM can accomplish, without necessitating a loop of prompt refinement that leaves me feeling like a worse developer for not just having thought through the problem myself, and resentful of the time wasted.

Edit: this entire thread about coercing Gemini into behaving is exactly the kind of crap I have zero interest in: https://news.ycombinator.com/item?id=44194061

it's really interesting to me because i feel like legacy code is where it actually excels very well. i have a large mental map in my head, so i usually just feed it some vague notion of what i want it do and use existing files as a reference ("i want to build this feature, look at X, Y, Z for reference. approach it this way" if i have some notion of how i want it go).

i usually don't let it go run off on its own unless it's a very defined task that i can review quickly later, i just review and approve every change and it takes big cognitive load off for me. at some point maybe this doesn't feel like "programming", but then i'll just tweak something else or modify it and then go onto the next review. i find i can't have it produce the entire thing and then review it since i have no idea how it got to where it did or takes just as much time to understand. but doing it this way it's faster + i gain understanding.

the prompts aren't overly complex or take a lot of time, certainly way less than speccing something out for a junior. all i have a is a base file for style and structure and then i describe the general problem and reference files ad-hoc. where i find it actually fails a lot is in novel code because it has nothing to ground it and starts exploring random stuff. i only use it for novel exploration to see what approaches it comes up with.

still trying to understand why there's this huge chasm between the two viewpoints. like a lot of the things you just said i can't resonate at all with. like maybe the 20% i feel like i'm "fighting" the LLM i just stop and go in myself. does that suck? sort of, but it's certainly way less tedious than directing some other person to do it or the time saved had i not used it at all.

edit: but to your point, yeah it really is just like magic with no way to like actually direct it in a way you would where a human would learn. maybe over a year ago i tried and wrote anything AI off beyond basic co-pilot completions (same issues, "fighting" the AI, having to specify a tons of exceptions in some god awful file). the new agents changed everything for me, esp claude code. i think it will only get better, so it's best to pick it up.

my only fears are

1) no juniors being trained, thus no future seniors. part of the power is that you have experienced people using it to enhance their context or understanding. for those with no experience and no drive to "improve" (honestly, think of the avg dev at big co) or straight up "vibe coding" i shudder at the output.

my hypothesis we are now going to enter a period where a LOT of shitty code is going to be created. it's already happening in education with people just cheating w/o learning. i already had issues trying to hire people who were using AI to get past initial exercises but failing on complex issues because they were just probably copy and pasting everything. best time to be a nimble startup.

2) top-down mandates to use this stuff. you should only use it when you want and if it helps you. i think there's this element of companies buying into the hype 110% and that puts a bad taste in everyone's mouth. "all devs replaced in 1 year!" type stuff.

Hmm, I should say that in the legacy case, it’s only been with legacy I’m already (painfully) familiar with, and knowing how difficult that code has been before to disentangle or to carefully stack another card on the already teetering house, I know upfront it’ll be a waste of time to let the LLM try.

In legacy that I don’t yet understand, maybe I am missing something in using a model with all that code in its context as an aid to build my understanding. I just cannot imagine handing off responsibility of modifying that code to the mystery machine. Tedious as it might be, I’m firmly of the view that I’d do myself and my team a disservice if I introduce changes I don’t fully comprehend, in fear that it’d only bite us later.

That fear comes from a couple cases of being bitten by changes made by other supposed seniors where I had my suspicions they let the LLM do the work for them, and went against my judgement in accepting their assurances that they’d tested everything worked.

Although, OK, for the legacy case, perhaps I should loosen my embargo in legacy frontend code where there’s a tight enough ‘blast radius’—meaning, the damage of a bad change is constrained to the bit of UI in question not working. Especially if that’s back-office frontend code where I have responsive users who I know will let me know if I broke something (because they surely know the warts of the system inside out, unlike me). In mission-critical backend code? Not a chance.

On the question of ‘fighting’ the LLM, maybe I do need to loosen up in how I want something done. In fairness I’m much more tolerant of that in a human, because a) there’s just often more than one way to do things; and b) with some devs I know there would or could be a fight that just isn’t worth it.

Which does come to an interesting point about ego and ownership, that if I regard the LLM code less as ‘my own’ perhaps I’d be more forgiving of its contributions. Would honestly make a difference if it’s not got my name on the commit.

Also comes back to one of the cases where I got burned. If the other dev hadn’t put their name on code which I’m sure was not theirs, then we could have had an honest discussion about it, and I could have better helped figure out whether what seemed to work was really trustworthy. Instead I had to weigh challenging them on it, and the awkward implication that I didn’t think they could have come up with the code themself.

To your two fears, totally agree, and undoubtedly some variant on these two cases has put a very bad taste in my mouth. Witnessing a junior essentially cheat their way through their final project for school, making a mockery of the piece of paper they got at the end. Being a victim not quite of a top-down mandate—somehow worse, with an exec head-over-heels bought into the hype, thinking they could lose a chunk of expensive headcount to no ill effect; not firing people, just making the situation miserable enough that people quit.

Reply on news.ycombinator.com