Comment by jascha_eng | Hacker News Reader

Comment by jascha_eng | Hacker News Reader

Parent Full thread

My impression so far is that the parallel agent story is a fabrication of "ai influencers" and the labs themselves.

I might run 3-4 claude sessions because that's the only way to have "multiple chats" to e.g. ask unrelated things. Occasionally a task takes long enough to keep multiple sessions busy, but that's rather rare and if it happens its because the agent runs a long running task like the whole test suite.

The story of running multiple agents to build full features in parallel... doesn't really add up in my experience. It kinda works for a bit if you have a green field project where the complexity is still extremely low.

However once you have a feature interaction matrix that is larger than say 3x3 you have to hand hold the system to not make stupid assumptions. Or you prompt very precisely but this also takes time and prevents you from ever running into the parallel situation.

The feature interaction matrix size is my current proxy "pseudo-metric" for when agentic coding might work well and at which abstraction level.

This is exactly my experience as well. The feature interaction matrix is growing as models get better, and I tend to build "prompt library components" for each project which saves time on "you prompt very precisely but this also takes time".

But so far that doesn't change the reality - I can't find any opportunities to let an agent run for more than 30 minutes at best, and parallel agents just seem to confuse each other.

Same. The only situation when I've consistently gotten a system to run for 20+ minutes was a data-analysis with tight guardrails and explicit multi-phase operations.

Outside that I'm juggling 2-3 sessions at most with nothing staying unattended for more than 10 minutes.

idk I haven't really hit the point with any llm that it comes up with useful abstractions on its own unless those abstractions have been in the training data.

E.g. imagine building a google docs clone where you have different formatting options. Claude would happily build bold and italic for you but if afterwards you add headings, tables, colors, font size, etc. It would just produce a huge if/else tree instead of building a somewhat sensible text formatting abstraction.

Tbf I wouldn't actually know how to build this myself but e.g. bold and italic work together but if you add a "code block" thing that should probably not work with font color and putting a table inside that also makes no sense.

Claude might get some of these interactions intuitively correct but at some point you'll have so many NxM interactions between features that it just forgets half of them and then the experience becomes sloppy and crashes on all edge cases.

The point of good software engineering is to simplify the matrix to something that you can keep arguing about e.g. classify formatting options into categories and then you only have to argue and think about how those categories interact.

This is the kind of thing LLMs just aren't really good at if the problem space isn't in the training data already => doing anything remotely novel. And I haven't seen it improve at this either over the releases.

Maybe this kind of engineering will eventually be dead because claude can just brute force the infinitely growing if/else tree and keep it all in context but that does not seem very likely to me. So far we still have to think of these abstraction levels ourselves and then for the sub-problems I can apply agentic coding again.

Just need to make sure that Claude doesn't breach these abstractions, which it also happily does to take short cuts btw.

FWIW I’ve used LLMs to invent new things. Not super groundbreaking fundamental research, but they were able to use physics to design a device that didn’t exist yet, from first principles.

Pics or it didn't happen

More seriously, what in the world "novel" physics device did you invent?

I didn’t say “novel physics” or “physics device”.

Okay, so rereading it as pedantically as you seem to insist:

You "invented" ("Designed") a "device" "using physics", and nobody has designed that "device" before, making it novel.

"From first principles" is a fun statement because people like Aristotle also thought they were reasoning from "first principles" and look how far it got them. The entire point of science is that "first principles" are actually not something we have access to, so we should instead prioritize what literally happens and can be observed. It's not possible as far as we know to trick mother nature into giving us the answer we want rather than the real answer.

Did you ever actually build or test this "device"?

Would you share a bit more?

Reply on news.ycombinator.com