HNHacker News
TopNewBestAskShowJobs

airbai

1 karma · joined October 10, 2011

submissionscomments
airbai··on Weak models write tool calls as prose. We started executing them
Author here.

The part I expect people to push back on is materialising prose into real tool calls. It looks like exactly the kind of clever thing that bites you later, and I went back and forth on it for a while.

What changed my mind was watching a user hit "continue" three times and get three identical blocks of text with zero executions. The model had already chosen the right tool and written complete, correct arguments. The only thing wrong was which channel it came out of. Refusing to honour that was costing a real turn to protect a purity that bought nothing — the alternative wasn't "the model does it properly", it was "the user gives up".

The constraints that make me comfortable with it: only tool names we declared to the model on that same turn, placeholder arguments are treated as a plan and never materialised, first occurrence only, and approval gates apply exactly as they would through the normal channel. It changes the transport, not the policy.

The detection bug underneath it is the more broadly useful lesson, I think. Our honesty check fired on "zero tool calls AND empty body", which quietly assumes a model that did nothing will also say nothing. That assumption fails precisely on the models where you need the check. The expensive failures all came with a confident paragraph attached.

Happy to go into any of the shapes we've seen — the fenced bash variant was the nastiest, because the reply ended with a question and collected an exemption we'd added for legitimate clarifying questions.

airbai··on Chrome Multitask Mode
Happy First April to everyone!
airbai··on In China, millions make themselves at home in caves
Actually, the cave you saw is very near my hometown, it is not that bad. The ventilation in it is good, and currently we have modern household electrical appliances as well.

Lots of local people adopted all things that exists in normal buildings, such as TV cable, Internet access, cars parked their own in the yard, etc.

You can refer to photoes at http://www.tripadvisor.com/Hotel_Review-g528735-d1307902-Rev...

airbai··on Web Quest
Is this site for 2011.11.11? Not sure what it is...