Investigate Heterogeneity Before Explaining It

Begin with the study records

Create an audit table that pairs every effect estimate with its population, intervention, comparator, outcome definition, follow-up window, design, and source. Differences in these fields provide a more useful starting point than searching for a single threshold that decides whether studies belong together.

First check avoidable inconsistencies: reversed group order, incompatible units, different outcome directions, duplicate cohorts, and SEs entered as SDs. Correct verified extraction errors with an audit trail. Do not relabel an inconvenient but valid result as an error merely because it increases variation.

Separate a diagnostic from an explanation

Cochrane's heterogeneity guidance distinguishes clinical, methodological, and statistical variation and emphasizes careful interpretation, particularly with few studies. A heterogeneity statistic describes a feature of the collected results; it does not identify its cause.

Inspect the forest plot's estimates and intervals alongside the audit table. Record whether studies differ in direction, magnitude, or precision. Similar point estimates with wide intervals tell a different story from precise estimates pointing in opposite directions, even when a short summary label looks similar.

Check a fictional Q-based calculation

For two independent estimates of 1 and 3, both with SE 1, the fixed-effect weighted mean is 2. Cochran's Q is 1 × (1 − 2)² + 1 × (3 − 2)² = 2, with one degree of freedom.

The conventional Q-based I-squared formula, max(0, (Q − df)/Q) × 100, gives 50%. The inverse-variance tool can illustrate this diagnostic. With only two fictional studies, that number is highly uncertain. It is not the proportion of participants affected, a percentage difference between treatments, or proof of a specific biological mechanism.

Trace any investigation back to the plan

Use the meta-analysis weight audit worksheet to record questions prompted by the results and which were prespecified. Examine plausible differences such as dose, follow-up, or design only where the data and review question support them. Avoid testing every available study characteristic and presenting the most attractive pattern as established explanation.

A low or zero I-squared estimate does not prove that effects are identical. Conversely, a high value alone does not tell you whether the absolute variation is clinically consequential. Consider effect scale, precision, study count, and the wider evidence.

If changing the synthesis approach, document the rationale and its effect on interpretation. Random-effects modeling does not repair incompatible questions or eliminate uncertainty about between-study variation. The final audit should separate corrected errors, unresolved comparability concerns, planned investigations, and exploratory observations. This preserves a defensible account of why the synthesis was retained, modified, or not performed.

Source checked on 2 November 2026; fictional Q and I-squared independently recalculated.