Audit Two-Rater Agreement Before Reporting Kappa

Define the rating exercise

Specify the items being rated, the two raters, the category definitions, and whether each rater works independently. Record the training procedure and the information available at the time of rating. Consensus decisions reached after discussion should not be substituted for the initial independent ratings when assessing agreement.

Create one paired record per item. Keep missing or indeterminate ratings visible and specify their handling before analysis. If participants contribute several items, such as multiple images, an ordinary independent-item analysis may not capture all the dependence in the data.

Construct the complete table

Label rows with rater one's categories and columns with rater two's categories. Confirm that labels appear in the same order so that diagonal cells represent agreement. Retain the category margins: they matter for interpreting how often each rater uses each label.

The scikit-learn documentation for Cohen's kappa gives kappa as (observed agreement − expected agreement)/(1 − expected agreement), with expected agreement calculated from the raters' category proportions. This reference quantity is part of the statistic's model; it is not a direct observation of raters guessing.

Check a fictional example

For 100 fictional items with two categories, suppose the table rows are 40, 10 and 20, 30. Observed agreement is (40 + 30)/100 = 0.70. Row proportions are 0.50 and 0.50, while column proportions are 0.60 and 0.40. Expected agreement is 0.50 × 0.60 + 0.50 × 0.40 = 0.50, giving kappa 0.40.

Use the kappa tool to verify the arithmetic, and retain the original four counts. A lone kappa value would conceal both the 70% observed agreement and the asymmetric pattern of disagreements.

Review the interpretation and design

Complete the rater agreement worksheet with the intended use and a justified acceptance criterion. Avoid treating conventional adjective bands as universal evidence that a rating process is adequate. Category prevalence and the raters' marginal distributions affect kappa; a high overall agreement can coexist with a less impressive kappa.

Unweighted kappa treats all disagreements alike. Ordered categories may justify a prespecified weighting scheme, but that is a different calculation from the binary unweighted example. More than two raters or repeated ratings require a suitably chosen method. If expected agreement equals one, the usual formula is undefined.

Report the table, observed agreement, kappa method, uncertainty where available, and missing-rating handling. Keep claims of agreement separate from validity: two raters can consistently make the same incorrect classification.

Source checked on 1 November 2026; fictional arithmetic independently recalculated.