The Reviewer Asked: Address Multiple Comparisons

Begin with the claim, not the correction button

A reviewer asking about multiple comparisons needs to understand how many opportunities the analysis provided to make a positive claim. List the outcomes, time points, comparisons, subgroups and models that support the conclusion. Then distinguish the primary confirmatory question from secondary and exploratory analyses.

The relevant family of hypotheses depends on the claims being made. Counting every number in a table is not a substitute for defining that family. Descriptive summaries and formal tests serve different purposes.

A fictional example

Imagine a study with five outcomes. The original protocol names one primary outcome; the manuscript highlights whichever outcome has the smallest p-value. Even if every calculation is correct, that presentation changes the apparent purpose of the study. The first revision should restore the planned primary question and label the other findings appropriately.

For illustration, if five independent null hypotheses are all true and each is tested at 0.05, the chance of at least one false rejection is 1 minus 0.95 to the fifth power, approximately 22.6%. Independence and true null hypotheses are assumptions of this calculation. It is not an estimate of the false-positive probability in your particular study.

Recover what was actually planned

Find the dated protocol and statistical analysis plan. Identify the primary endpoint, any hierarchy, the intended testing family, the chosen error criterion and the planned adjustment. Record amendments and when they occurred relative to access to the results.

If there was no prespecified multiplicity strategy, say so. A newly chosen method can improve the revision, but it cannot retroactively make a decision prespecified. Avoid trying several adjustments and reporting only the one that preserves a preferred result.

Choose an approach that matches the objective

Family-wise error control and false-discovery-rate control address different error criteria. Their suitability depends on the study purpose and the intended claims. A hierarchy, a simultaneous testing procedure, or an explicitly exploratory presentation may be appropriate in different circumstances; no single adjustment is universally correct.

The FDA guidance on multiple endpoints discusses grouping, ordering and testing endpoints in drug trials. Its regulatory scope should not be presented as a universal rule for every observational analysis. Use the relevant study framework and statistical advice.

Show what changed

A useful response identifies the hypothesis family, explains the method and timing of the decision, and points to the revised methods and results. Report estimates and uncertainty alongside the testing results. Make clear whether confidence intervals are ordinary individual intervals or were constructed to provide simultaneous coverage; adjusted p-values do not automatically turn ordinary intervals into simultaneous intervals.

Do not claim that adjustment proves a finding true or that a non-significant adjusted result proves no effect. Discuss how the revised analysis changes the strength and scope of the conclusion.

Use the Reviewer Comment Decoder and response tracker to organize the response. For an existing manuscript, statistical revision support can help align the methods, tables and claims.