Show HN: DFAH-Bench – same agent decision, different tool paths

Hey - released this PyPi library after getting annoyed most agent evals only focus on outcomes, and the tool calling path is just as important in recognizing the outcome decision. So defined decision agreement rates, trajectory agreement rates etc etc

Most tell us whether the final answer was right, but not whether the agent actually did the same thing twice.

In prospective replays of financial services tasks, agents reached the same decision about 94–95% of the time. Exact tool-name paths matched only 67–69%. Once the tool arguments were included, agreement fell to 45–52%.

I don’t think every different path is automatically a failure. Sometimes it’s harmless flexibility. But if an agent can touch files, APIs, email, or financial controls, I’d at least like to know that it changed its path before calling the system stable.

The package repeatedly runs your agent, stores the traces, and compares the decision, tool order, arguments, and result identities.

Quick local test - no key or paid model needed:

dfah check-agent --agent dfah.demo → dfah check-agent --agent dfah.demo:toy_agent --episode-timeout-s 5 Paper: https://arxiv.org/abs/2607.20491 PyPI: https://pypi.org/project/dfah-bench/

It’s still early. I’m especially curious about two things:

How would you separate harmless path diversity from instability that should block or flag a run?

And which agent framework should I build the next adapter for? Or is there other ways folks are thinking about tracking tool paths and decision agreements