LLMs are increasingly being used as evaluators — grading outputs, ranking options, and even guiding autonomous agents. But raw model scores are unstable: they drift over time, vary across models, and don’t map linearly to real quality.

This write-up explores why LLM-based judging is unreliable out of the box, how calibration curves can improve stability, and why multi-model consensus may be necessary for trustworthy evaluation systems.

Curious how others in the industry are handling calibration, drift mitigation, or cross-model agreement for “LLM-as-a-judge” setups.