A novel open-source framework for evaluating conversational agents

this one is really a game changer:

This is how it works - the framework is organized into these powerful components:

1) Policy Graph Builder - automatically maps your agent's rules 2) Scenario Generator - creates test cases from the policy graph 3) Database Generator - builds custom test environments 4) AI User Simulator - tests your agent like real users 5) LLM-based Critic - provides detailed performance analysis

It's fully compatible with LangGraph, and they're working on integration with Crew AI and AutoGen.

They've already tested it with GPT-4o, Claude, and Gemini, revealing fascinating insights about where these models excel and struggle.