Author here. I left in all those shipping changes to the agent - a prompt modification, a more lenient guardrail, some increased temperature -that altered the behavior of the agent without altering its execution, which flew past code reviews and CI and only came to light in prod.
ctxwitch diffs changes made to the prompt/model/tools/RAG/memory/guardrails of an agent and categorizes their behavioral effect in 12 different ways at 5 levels (cosmetic -> breaking). A removal of guardrail or reversal of any rule gets marked as Breaking, and CI system gets notified that a more rigorous evaluation is needed, while cosmetic modifications sail through. It works both locally and on Github actions, takes about 100ms, deterministic (no agent executions, traces, or LLMs are used; there's an option to use LLM judges for truly subjective cases).
But in an attempt to address the obvious problem: a configuration difference doesn’t prove something was actually run, but rather that there was intent to make it happen. Therefore, I designed it as a triage/gate - "this is something worth looking at".
pip install ctxwitch
witch tour #disposable sandbox, no api
Apache 2.0. I really would like to know for those running ai agents in production what diffs have caught up with you that something a diff like this should have spotted, and what more would it need to pickup to get into your CI?