Evaluate Interval Coverage and Width Together

Define the coverage event

For each simulated dataset, an interval covers the target when its lower endpoint is no greater than the true value and its upper endpoint is no less than that value. Record the interval method and nominal level. Coverage is a repeated-sampling property assessed across replicates, not the probability that a fixed realized interval contains the parameter.

The rsimsum documentation includes interval coverage and Monte Carlo error in simulation summaries. Keep the true value associated with the correct scenario so a shifted effect scenario cannot accidentally be assessed against zero.

Check interval validity first

Verify endpoint order, finite values and compatibility with the effect scale. Record missing or failed intervals explicitly. Decide how those failures enter the performance report rather than treating them as if they never occurred.

Pair coverage with width or another precision summary. An extremely wide interval can achieve high coverage while providing little useful information. Conversely, a narrow interval can look attractive while missing the truth too often.

Calculate a hypothetical result

Suppose 950 of 1,000 valid intervals contain the scenario’s true parameter. Estimated coverage is 95%. If another method covers in 970 of 1,000 but its median width is twice as large, the higher coverage alone does not establish a better practical method.

Now suppose only 900 of 1,000 attempted fits return valid intervals and 855 of those cover. Coverage among valid intervals is 855/900 = 95%, while only 855/1,000 attempts produce a valid covering interval. Report the 10% failure rate separately and make the denominator explicit; the two proportions answer different questions.

Inspect patterns of missed coverage

Separate misses below and above the truth when asymmetry matters. Examine coverage by scenario rather than pooling easy and difficult conditions into one reassuring average. Record Monte Carlo uncertainty so small deviations from the nominal level are not overinterpreted.

Preserve representative intervals and diagnostics for failures or extreme widths. Check whether a method’s nominal level and critical values match its stated construction. The final report should let readers evaluate calibration, precision and reliability together, with no silent exclusion of datasets on which the interval procedure could not deliver a usable answer.

Source verification: 6 November 2026. Examples are hypothetical.