Interesting approach! I’ve been building something complementary on the deterministic side.
LLM-as-judge guardrails are fundamentally probabilistic and can be gamed or hallucinate themselves (as several comments pointed out).
That’s why I built EvalView — it does full trajectory snapshots + diffs so you can see exactly what changed, plus a lightweight zero-judge model-check that directly pings the model and reports drift level (NONE / WEAK / MEDIUM / STRONG).
Gives you deterministic regression detection that works alongside (or instead of) LLM judges.
https://github.com/hidai25/eval-view
Curious how you handle drift detection in CrabTrap.