You bought the eval tools. Do you have the examples that matter?
Models, frameworks, and metrics all rest on labeled examples. If those examples do not resemble real users, a polished score will not make a sound decision.
A golden set is not a folder of convenient correct answers. It is a versioned sample of real tasks, risks, and decision boundaries, and it determines whether downstream metrics mean anything.
Sample the real distribution
Stratify real requests or tickets by task, language, length, user group, and risk. After privacy review, retain difficult, ambiguous, and failed cases rather than only clean examples.
Define evidence beyond one answer
Some tasks have a single label; others need a rubric, required facts, and unacceptable errors. Store provenance, validity date, annotators, adjudication, and acceptable answer range.
Split development and holdout sets
Use a development set for iteration and a holdout set for release gates. Add production failures deliberately by deciding whether they represent common traffic or rare risk.
Pre-release checks
- The set covers traffic mix and high-risk tails
- Annotation disagreement is recorded
- Every evaluation identifies the dataset version
Sources
Related
Writing, extraction, long reasoning, and high-volume support do not need the same model. Cost, latency, privacy, and reliability rarely point to one name.
GPUs, operations, inference tuning, and downtime all land on the same ledger. Self-hosting only makes sense after you have done the math on volume and people.