Hacker News Reader

Cool.. but I still don't get how this is going to save money. It seems to me that it might actually burn more money just because the whole system now seems to be coming from different LLMs.

Also, small LLMs are prone to stop before completion, throw errors and produce loops. Is this factored in the design of the tool? I am not sure.

edit: spellcheck

View on HN

It saves money because some agent sessions can be entirely handled by a smaller model (also relevant: subagents use fresh context windows so a subagent with a simple task can be routed to a smaller model even if the main agent needs a frontier model).

Totally right about small LLMs btw, that's why we trained this on real agent sessions where we forced it to use different models. If the routing model sees small models can't handle a certain type of task then they won't be assigned. (Also as a fallback we have some guardrails that will have a bigger model come in to "rescue" a smaller model if it gets stuck)

Reply on news.ycombinator.com