Hacker News Reader

Comment by jdiff | Hacker News Reader

I think the point being made is that when you're feeding outputs into inputs, both 93% and 90% rapidly collapse to 0% without some way to recover.

View on HN

Which is precisely what I'm responding about above. 93% actually collapses much slower than 90% since the rate of reduction is based on that of all of the previous rounds, not just the current chance. E.g., this is the difference between 14 rounds before an expected failure instead of 10. An extra 4 rounds deep at the same error rate is a big step forward despite a small change in the base percentages. Add just few more percentage points again and you've more than doubled the depth.

I.e. it's precisely when you look at these chained that the difference is actually very large. It's really only the 1st turn or two of prompts where the difference seems negligable.

How are you running that math and what are you consider to be "expected failure"? If even 30% of your runs go into the garbage, that sounds horrifically unreliable, and both 7 and 10% failure rates hit those margins in less than 5 iterations.

Not to mention that the deeper your iterations, the more money burned on garbage before you realize that data in turned to garbage out turned to garbage in somewhere in the chain.

I'm taking 0.90^n and 0.93^n and seeing 0.93 does more than the initial 3% better over time, which preciesly saves you from things like spending tokens on garbage by this larger factor. You're free to pick whatever target reliabilities from that. E.g. you can maintain above 80% accuracy over 3 turns rather than 2 or 50% over 6 turns instead of 3 or you can pick another percentage point and see how the depth to expected error/expected error at depth numbers are much better than 3% and how another 3% actually doubles accuracy because this is all non-linear.

There's also the matter of "90% compared to what". If it was 90% per attempt compared to a calculator then I'd just use a calculator. If it was 90% per attempt for something I have a 10% chance of getting right per attempt then it's not really so bad. OTOH if it's 90% per attempt at keeping my car in the lane then it's obviously quite bad. I.e. there isn't one "reliable percentage" to target which is why I'm comparing the curve instead of just the most stringent things I can think of in isolation.

Completely agree that error tolerance depends on what it is that we're doing. Some things demand no errors at all, some things have a cost to attempt, some things you can't pre-evaluate for correctness if you don't already have the prerequisite knowledge and you only get one chance. Not everything is software where we can conjure 50 more end-to-end tests to ward off demons. When it's not a harmless throwaway weekend project, 7% is already a huge error rate before you ever think of feeding it back into itself.

I of course agree not everything is fine with 93%, but via the very same reasons believe a 3% error rate change is a big deal rather than a reason to dismiss the improvement: not every task has the same tolerance, but there's not many percentage points left until it'd be claiming all of them.

That's completely orthogonal to whether you can still find tasks which require more accuracy or not (except for the sole case someone claims it's 100%).

You might be thinking along the lines of the O-ring theory. Indeed long supply chains accumulate errors exponentially.

However the origins of theory explored the outcome of disastrous and irreparable outcome of space shuttle exploding, lives lost, and billion of dollars destroyed.

When LLM goes off the rails the cost is much lower, and retry is on the table. In fact our core competency shifted from writing code to creating environment where agents can retry cheaply and course correct based on feedback.

A modest increase in the number of turns it takes to reach decorehenve could make a difference between Decoherence and reaching the a grounded checkpoint.

If failure is essentially guaranteed somewhere in that chain, success becomes impossible. You don't need to lose lives to have simply have a fruitless endeavor. And when we're talking about law, we're talking about more than a fruitless endeavor. Law also isn't software that can have tests crafted and fed back into the loop.

Reply on news.ycombinator.com