Hacker News Reader

hidai25on CrabTrap: An LLM-as-a-judge HTTP proxy to secure agents in production

You’re right that I mixed runtime enforcement with CI drift/regression testing. Different layer, different job.

I meant it as complementary, not equivalent. CrabTrap for runtime control, EvalView for deterministic testing/diffing. My bad on making it sound like a drive-by promo.

hidai25on CrabTrap: An LLM-as-a-judge HTTP proxy to secure agents in production

Interesting approach! I’ve been building something complementary on the deterministic side. LLM-as-judge guardrails are fundamentally probabilistic and can be gamed or hallucinate themselves (as several comments pointed out). That’s why I built EvalView — it does full trajectory snapshots + diffs so you can see exactly what changed, plus a lightweight zero-judge model-check that directly pings the model and reports drift level (NONE / WEAK / MEDIUM / STRONG). Gives you deterministic regression detection that works alongside (or instead of) LLM judges. https://github.com/hidai25/eval-view Curious how you handle drift detection in CrabTrap.

hidai25on Show HN: EvalView – Catch agent regressions before you ship (pytest for agents)

Your agent worked yesterday. Today it's broken. What changed?

  EvalView catches regressions before they hit prod. Tool changes, hallucinations, cost spikes.

  Save a baseline. Run evalview run --diff. CI fails if behavior drifts.

  pip install evalview && evalview demo

  No API key needed. Ollama support for free local evals. Chat mode if you hate memorizing CLI flags.

  Built this after an agent started inventing numbers in production. Would love feedback from anyone shipping agents.

  Hidai

hidai25on Show HN: EvalView pytest style tests for AI agents (budgets, hallucinations)

Hi HN, I built EvalView after an agent that worked fine in dev started inventing numbers in prod. Tracing showed me what happened after the fact, but I wanted CI to fail the deploy the moment the agent drifted. EvalView is basically pytest for agents: you write a YAML test, run it N times (to catch flakiness), and fail the build if behavior regresses. Example:

```yaml name: "Refund policy doesn't hallucinate" runs: 10 pass_rate: 0.8 input: query: "What's our refund policy?" assert: - tool_called: "kb_search" - no_unsupported_claims: true - max_cost_usd: 0.05 ```

Instead of exact text matching, the checks focus on constraints: did it call the right tools, did it make claims not supported by tool results/context, and did it stay within cost/latency budgets. I also added optional local LLM-as-judge via Ollama so evals don’t burn API credits on every run. If you’re shipping agents to prod, what’s been your worst failure mode: tool misuse, budget blowups, or confident nonsense? Happy to answer questions. Hidai