Your model scored 85%. Why is production still a mess?
One score is convenient, but it hides errors in retrieval, format, facts, and user experience. Split the work apart to see where the model actually fails.
Pass@1 asks whether the first attempt passed an automated check. It does not show severity, stability, or the work users spend correcting an answer.
Split quality into diagnostic dimensions
Score task completion, evidence, format, style, and safety separately. Extraction needs field-level precision and recall; RAG needs separate retrieval and answer-faithfulness measures.
Measure stability and severity
Repeat critical samples and record variance. A wording difference and a wrong payment amount are not equivalent failures; give critical errors their own gate or human approval requirement.
Use production signals to extend offline evaluation
Feed retries, edits, abandonment, human takeover, and complaints into the evaluation set. Production signals reveal blind spots but still require sampled human review.
Pre-release checks
- Metrics identify the failing stage
- Averages cannot hide critical errors
- Offline and production signals are reconciled
Sources
Related
Writing, extraction, long reasoning, and high-volume support do not need the same model. Cost, latency, privacy, and reliability rarely point to one name.
GPUs, operations, inference tuning, and downtime all land on the same ledger. Self-hosting only makes sense after you have done the math on volume and people.