Report Monte Carlo Precision for Simulated Probabilities

Define success within a simulation repetition

A simulated probability might represent confidence-interval coverage, rejection of a hypothesis, or successful model convergence. Write an explicit success indicator for one repetition and keep its definition fixed across the scenario. The number of simulated datasets is different from the number of participants generated within each dataset.

Morris, White, and Crowther's simulation-study tutorial emphasizes reporting Monte Carlo uncertainty when performance is estimated from finitely many repetitions. Increasing repetitions improves numerical precision for the chosen scenario; it does not establish that the scenario represents a real population.

Keep failed repetitions visible

Record attempted repetitions, completed analyses, failures, and the denominator used for each performance measure. A coverage calculation conditional on successful fits can differ from performance across all attempted fits. Do not silently remove failed runs and describe the result as unconditional performance.

Check whether repetitions are independent under the random-number design. Reusing correlated draws or treating dependent chain iterations as independent Bernoulli repetitions changes the uncertainty calculation. Preserve seeds, streams, and scenario labels in the simulation record.

Recompute a fictional probability

Suppose 950 of 1,000 independent repetitions satisfy a specified coverage criterion. The estimated coverage is 0.95. Its plug-in Monte Carlo SE is sqrt(0.95×0.05/1,000), approximately 0.006892, or 0.689 percentage points.

The Monte Carlo precision tool also supplies a 95% Wilson interval for the simulated success probability. For these invented counts it is approximately 93.47% to 96.19%. This interval describes uncertainty from the finite simulation run, not uncertainty in a treatment effect estimated within one of its datasets.

Plan repetitions for useful precision

If the success probability is unknown, p(1−p) is at most 0.25. A conservative planning count for target SE 0.005 is therefore ceiling(0.25/0.005²) = 10,000 independent repetitions. This is a precision bound for a Bernoulli proportion, not a sample-size calculation for a clinical study.

Use the simulation precision worksheet to record the scenario, success definition, failure handling, target precision, and planned repetitions. At the same success proportion, quadrupling repetitions halves the plug-in SE; small numerical differences between methods may require considerably more simulation than a broad comparison.

At zero or all successes, the plug-in SE is zero, but that does not prove exact knowledge of the underlying probability. The score interval remains informative about finite-run uncertainty. Document this boundary behavior rather than reporting a false impression of perfect precision.

Finish with the estimate, repetition count, Monte Carlo SE or interval, and the conditions under which the result was generated. A highly precise simulation estimate can still answer the wrong question if the data-generating mechanism, estimand, or method implementation is inappropriate.

Source checked on 4 November 2026; fictional precision calculations independently recomputed.