Comment by tstrimple | Hacker News Reader

Comment by tstrimple | Hacker News Reader

Parent Full thread

Any time the LLM can establish a tight feedback loop it can iterate effectively enough to make good progress. But if for one reason or another that feedback loop is long, the more guidance and direction they seem to need. If they cannot correct their course based on feedback from their own output they need an outside observer to push them towards different paths. CRUD stuff is the obvious easy answer. It represents some absurd amount of software development work and is drastically overrepresented in training data. For good and bad. Lately I've been working on a diffusion model for terrain generation. The goal is to be able to recreate realistic terrain based on water flow and erosion on top of plate tectonics for much longer geological time terrain adjustments. These are much longer feedback loops. Sometimes it'll spend 6+ hours training a diffusion model on my 5090 just for the quality to degrade compared to the last experiment. So letting the model go wild trying different approaches and evaluating the results doesn't work nearly as well as when it can iterate 1000 times in those 6 hours while checking its own answers.

Any time the LLM can establish a tight feedback loop it can iterate effectively enough to make good progress.

Not any time. Your feedback loop must provide some direction so the iteration is at least heading towards an improvement. For novel applications you might not have a convenient way to do that yet.

Even with that provision there's still a possibility of incrementally updating output converging on a local maximum. That local maximum might not be a solution at all. If it is a solution then the model might still miss much better solutions that a real developer with more insight might find.

As you said yourself the models need external feedback if they don't have sufficient information to make adjustments by themselves. In some cases that feedback might be "Start again because you're hopelessly off track". Sometimes even when they start again they just get hopelessly off-track again. That can happen even if they are explicitly prompted to do something else. So even closely supervising the LLM and not using self-iterating agents isn't guaranteed to get good results for novel applications.

Maybe I'm overly pedantic, but in my mind a feedback loop involves both feedback and a loop. So by my reckoning the "feedback loop" must, by definition, guide the direction of the next step. I hold this true of feedback loops with zero LLM tooling involved. It's the same thing. The "entity" determining whether to continue the loop or not must have a good enough definition of success or improvement. Of course there is a lot of variability in how effectively the feedback is delivered. That's one of the variables the human setting up the loop is responsible for establishing in our brave new world.

You're right though that in new territory you may often have domain experts who still cannot adequately define success in an automated fashion because this is literally something that hasn't been created before. So they must be a part of the loop. They cannot necessarily put into code and descriptions what "better" or "right" is for an LLM to follow. But they can recognize it when they see it. In this case, it's still best to create the tightest feedback loop you can. If the LLM is waiting 2 days for your feedback, the loop is 2 days long.

I think it depends on what you count as feedback.

I would say that a system with some sort of binary evaluation that indicates whether it succeeded or not is receiving feedback. It could also be some kind of sliding scale - how close did it get?

Either of those might be useful for determining a good future direction to iterate in but that would surely depend on the nature of the problem and how it's possible to search for solutions. I'd still call it a feedback loop even if the evaluation wasn't enough to point in a specific direction for a further iteration though. YMMV.

Reply on news.ycombinator.com