viksit

> heavily-customized and terminal-heavy setup

this exactly. most people can’t set it up that well.

underrated comment haha. made my day

I think the point here is the bot didn’t come up with the idea on its own, but was seeded by a human with an entrenched interest / experience in that space. The PR around Moltbook has been that Agents are coming up with these ideas through some sort of swarm interaction with each other, and developing novel, emergent capabilities — which is not true.

this is a great question. what are the main use cases that you have for this? i’ve been working on a library for something similar and exposing it via an mcp interface. would love to pick your brain on this (@viksit on twitter)

Yes! thanks for the memory haha.

you definitely succeeded in your humorous endeavors ;) i snorted haha

In my fourth post in the series, I tackle how to make multi-step agent workflows learn behavior from data. Most agents today rely on vibes: prompt tuning, hand-written templates, and hope(!). This post is about replacing that with metrics and optimization.

Each branch in the workflow learns how to behave, not just where to route. I show how to set up a reward, plug in an optimizer, and treat agent behavior as something you can tune like a model.

mostly aligned on this. couple of thoughts:

- raw accuracy is now a "vanity" metric. so the benchmarks need to get more sophisticated, and i think they're going to have to be far more task specific than hotpot or hover. they've become like the mnist of multi hop.

- in my use of MIPROv2 and SIMBA, I see a fair amount of improvements for multi hop tasks (published some of these on hn before). I'm going to try GEPA and see how it performs. so I think we're at the start of what I would call "meta learning".. tuning across a huge search surface rather than tweaking one prompt. hyper param search for higher dim spaces.

- tokens burned should be a reported result

vJEPA models, lecun's approach towards world models that have been derided by a lot of naysayers. (personally I think thats the way to go)

they’ve already written one! see omar’s x account for details!

Based on the feedback from my last HN post on differentiable routing, I ran a follow-up benchmark: local RNN vs GPT-4o for tool selection in LLM workflows.

Same accuracy, 40% lower cost. Appreciate all the suggestions, this post builds on them.

Following up on my last post about optimizing tool selection with differentiable programming, I’ve been thinking about how to extend those ideas to full agent workflows. This post shares some early experiments using DSPy to optimize routing and structure end-to-end for a sample customer service agent workflow. Feedback welcome!

would you have a link?

there's a world where the model could infer that as well!

yes, AFAIK right now, there are no easy ways of "slimming" context because no one knows what it should be or how.

great q. thats coming up as a future post in the series.

this is really interesting! yes, its my substack.

also, if you're down, love to connect and talk more about what use cases / techniques you're using. I'm @viksit on X dms if that works.

for sure, there's a way here where I think we ought to be able to learn multiple tool calls and prompts together with real world data. investigating that next.

(author here, put the code in a gist here for reference)

https://gist.github.com/viksit/c67d1d960c4cec89488290496defb...

+1 thanks for mentioning MCP!

re: different tools (apis vs mcps). in my mind, there should be no real difference at what kind of tools is called at this moment since I model this as a softmax over a label set of tools.

that said, an idea I want to investigate is whether tools can live in a learned embedding space, where selection isn’t a softmax over discrete labels but a nearest-neighbor or attention mechanism over continuous vectors.

this is the intuition I'm developing as we speak and in some of my other comments on this thread (see differentiable state machine comment).

+1 - you can propagate the loss for a workflow across prompts + tools, which would make it much better to do resilient workflows. or "agents" as everyone calls them now ;)

this is my goal :) appreciate the feedback.

+1 - the biggest issue is not being able to fine tune the llm to learn the specifics of how to make a tool call better over time, which an approach like this can bring to the table.

+1 on the control flow point.

I think of an llm as a differentiable interpreter of a program. it should do decision making (tool selection, argument routing), branching logic via weights + gates etc.

so as a differentiable state machine:

- each state == a stage in your workflow

- transitions == tool calls

- encode this as a rnn or graph

and learn transitions and actions via supervision or RL

this is a great point, ty.

in my mind the biggest difference is llms that are invoked during a workflow, and llms that are invoked when _creating_ code (codegen).

for the former, tools could be well defined till they are small in number, but at some point, the system needs to examine a library of tools, understand how to call it and integrate it, and at its peak, even create new tools to talk to systems not already present in that library (codegen).

it’s not just about selection. say you’ve got 100k tool calls — in the current hosted llm setup, you don’t actually learn anything new about your data to improve future tool accuracy.

this gets worse when you’re chaining 3–4+ tools. context gets noisy, priors stay frozen and there's prompt soup..

my intuition here is: you can learn the tool routing and the llm prompts before and after the call. (can always swap out the rnn for a more expressive encoder model and backprop through the whole thing).

super useful when you’re building complex workflows -- it gives you a way to learn the full pipeline, not just guess and hope.

ty for the feedback, yes, balancing bad code blocks on substack vs making it look pretty lol. I'll post code next time.

yes apologies, the code rendering in substack wasn't great, but I'll put this in a gist!

thank you, appreciate the comment! thats a great point -- as I'm developing this intuition, I'm designing an eval which does a comparison of the openAI example there + tool call using a simple RNN + one that uses an encoder model. would love more feedback (on blog / X etc) when I post.

great point, appreciate the comment. totally agree with your framing, though i think there’s still a gap in how tool use is handled today.

quick note: it doesn’t have to be an rnn. i’ve got a follow-up example coming that uses a transformer-style ToolController with self attention, more expressive routing, etc.

but here’s the thing — when you rely on few-shot bootstrapping the LLM, you never end up updating the model's priors. even after 100k tool calls, you’re still stuck in the same polluted context window and its all stateless.

this gets worse fast with more than 3–4 tool calls, especially when there’s branching logic (e.g., if api1 > 5, go left, else right).

what this approach offers is: backprop through tool calls. you can tune prompts and update priors across the full workflow, end to end. trying to develop this intuition a bit more, and would love feedback.

thanks for the suggestion on the eval — will post that comparison soon.