Hacker News Reader

I use promptfoo for our internal evaluations. There are a ton of different "assertions" (tests) that you can write, including model-graded evaluations using rubrics.

This is far from a solved problem, but there are options out there for systematic testing of LLMs.

Probably non-optimal long term, but you can easily whip up rudimentary tests with your LLM/lib of choice (see my other comment at https://news.ycombinator.com/item?id=42322794) without using any of the eval platforms.

Offline evaluators such as laminar and (evalite wip) stay out of the way fortunately. Evals can be very useful when they do.