TWO-RATER AGREEMENT PLANNING WORKSHEET
StatMedAnalysis | 1 November 2026

Purpose: Define the rating task and sampling design before interpreting agreement.

Instructions: Save a working copy. Complete each prompt using aggregate summaries or design decisions. Do not include patient identifiers, actual allocation sequences, or confidential records. Write NOT REPORTED, NOT APPLICABLE (with reason), or UNRESOLVED instead of silently leaving a field blank. Resolve flagged issues before using outputs in a report. This is an original planning aid, not a validated instrument or approval certificate.

1. Target population of items and intended use of the ratings:
Response: __________________________________________________________
Evidence/action: ___________________________________________________

2. Rater selection, training and independence/blinding:
Response: __________________________________________________________
Evidence/action: ___________________________________________________

3. Binary category definitions and ambiguous-case rules:
Response: __________________________________________________________
Evidence/action: ___________________________________________________

4. Both yes / rater1 yes only / rater2 yes only / both no counts:
Response: __________________________________________________________
Evidence/action: ___________________________________________________

5. Total items and missing/unpaired ratings; exclusion reasons:
Response: __________________________________________________________
Evidence/action: ___________________________________________________

6. Observed agreement and each rater marginal proportion:
Response: __________________________________________________________
Evidence/action: ___________________________________________________

7. Unweighted kappa and method; note if undefined:
Response: __________________________________________________________
Evidence/action: ___________________________________________________

8. Sampling variability and appropriate confidence-interval plan:
Response: __________________________________________________________
Evidence/action: ___________________________________________________

9. Category prevalence, asymmetric disagreements and potential consequences:
Response: __________________________________________________________
Evidence/action: ___________________________________________________

10. External validity/accuracy question (agreement does not establish truth):
Response: __________________________________________________________
Evidence/action: ___________________________________________________

11. Prespecified acceptability reasoning; avoid universal verbal cutoffs:
Response: __________________________________________________________
Evidence/action: ___________________________________________________

12. Data version, source, preparer, independent check and date:
Response: __________________________________________________________
Evidence/action: ___________________________________________________

Related guide and tool: https://www.statmedanalysis.com/resources
Keep the completed copy in your approved research workspace.