A green dashboard does not mean your LLM judge is right
A model can score confidently while disagreeing with people. Compare it with human labels before treating the result as a quality signal.
LLM judging can expand evaluation coverage, but the judge is itself a biased, drifting measurement instrument. Calibrate it against the people whose judgment matters.
Write an observable rubric
Split quality into observable dimensions such as factuality, task completion, format, and risk, with examples for each level. A single overall score can hide critical failures.
Calibrate with blind samples
Use samples with two human labels and adjudication, blind model identity and output order, and compare agreement, category errors, and missed critical failures.
Prevent judge overfitting
Keep a holdout set outside prompt tuning and add new failures over time. Recalibrate when the judge model, rubric, prompt, or evaluated model changes.
Pre-release checks
- Humans can independently apply the rubric
- Critical-error false negatives are tracked
- Judge version and prompt are traceable
Sources
Related
Writing, extraction, long reasoning, and high-volume support do not need the same model. Cost, latency, privacy, and reliability rarely point to one name.
GPUs, operations, inference tuning, and downtime all land on the same ledger. Self-hosting only makes sense after you have done the math on volume and people.