Methodology
Measuring expert consensus: Cohen's κ in plain English
How we quantify agreement between experts — and why raw accuracy isn't enough.
Quality Engineering·September 8, 2026·5 min read
When three attorneys label the same contract clause, how confident should you be in the result? Agreement statistics answer that question.
Beyond percent agreement
Two raters who always pick the most common label will agree often by chance. Kappa corrects for that, measuring agreement above what randomness would produce.
How we use it
We track κ per task, per expert and per bounty. Low agreement triggers adjudication by a senior expert, and the outcome feeds back into each contributor's Expert Score.