One Decision Changes the Study: Testing Subgroup Differences

Two p-values do not test a difference

Suppose a fictional trial reports an estimated treatment difference of 4 units in one subgroup and 3 units in another. The first subgroup has a smaller standard error and a p-value below 0.05; the second has a larger standard error and a p-value above 0.05. Those results test each effect against zero. They do not directly test whether 4 differs from 3.

The distinction matters when a paper concludes that a treatment works “only” for one group. Unequal precision can produce different significance labels even when estimated effects are similar.

Ask the interaction question

A subgroup claim concerns whether the effect changes across groups on a defined scale. A suitable analysis compares the effects directly, often using a treatment-by-subgroup interaction in an appropriate model. The model must respect the design, clustering, repeated observations, and uncertainty; a separate p-value for each group is insufficient.

For a numerical illustration only, assume independent subgroup estimates of 4 and 3 units, with standard errors of 1.5 and 2.0. Their estimated difference is 1 unit. Its standard error is the square root of 1.5 squared plus 2.0 squared, or 2.5. Under a normal approximation, the 95% interval for the difference is about −3.90 to 5.90 units. The example gives little precision about effect modification despite different within-group significance labels.

Do not use this independence formula when estimates share participants or have correlated errors. An appropriate model must account for that covariance.

Define the scale and timing

Effect modification can look different on absolute and relative scales. Specify which scale answers the scientific question. Also record whether the subgroup and its cut point were chosen before the results were examined. Searching many cut points and presenting the most striking result understates the amount of exploration.

Consider the biological or clinical rationale, number of subgroup comparisons, sample sizes, precision and consistency with other evidence. A non-significant interaction does not establish identical effects, especially when the comparison is imprecise.

Be particularly careful with meta-analysis

A comparison of studies with mostly older participants against studies with mostly younger participants is a study-level comparison. Other study characteristics may differ too. It is not equivalent to comparing age groups within randomized trials.

Cochrane Handbook Chapter 10, section 10.11 explains interaction, subgroup analysis and the pitfalls of comparing within-subgroup significance. It also emphasizes the risks of multiple subgroup analyses. Treat exploratory findings as hypotheses unless the wider evidence supports a stronger interpretation.

Rewrite the conclusion around the evidence

In the fictional example, a defensible account would report both subgroup estimates, the direct comparison and its wide interval, then state that the evidence is insufficiently precise to characterize a difference. It would not recommend treatment for one group while ruling it out for the other solely from the two p-values.

Explore effect estimates and uncertainty in the Results Interpretation Playground. For synthesis planning, use the pooling checklist; for your own analysis, Understanding Your Results can help connect the statistical comparison to the research question.