Playbook
Building a gold-standard evaluation set in one week
The playbook we use to take enterprise teams from scoping to a certified eval set in five days.
Solutions Team·August 11, 2026·8 min read
Most teams overestimate how long it takes to build a trustworthy evaluation set — and underestimate how much rubric design matters.
Day 1–2: Scope and rubric
Define the decisions the model must make and co-write a rubric with two domain experts. Ambiguity here is the biggest source of downstream disagreement.
Day 3: Calibrate
Run a 50-item pilot, measure agreement, and refine the rubric where experts diverged.
Day 4–5: Scale and certify
Route the full set to matched experts with 3× redundancy, adjudicate disagreements, and deliver with provenance metadata.