Report the Resolution of a Simulated P-Value

Identify the source of randomness

Some tests use a random sample of allowed permutations or simulated null datasets instead of enumerating the full reference distribution. Their reported p-value then has Monte Carlo variability in addition to the study’s sampling uncertainty. Record the number of draws and the counting convention.

The SciPy permutation-test documentation describes a conservative adjustment for randomized permutation p-values. Different implementations may use different conventions, so preserve the actual method rather than assuming every printed decimal has the same interpretation.

Define the exceedance event

Specify the test statistic, alternative, handling of ties and what qualifies as at least as extreme as the observed statistic. For two-sided tests, document the construction; simply doubling a tail probability is not interchangeable with every absolute-statistic approach.

Distinguish a randomized test from complete enumeration. When every allowed arrangement is evaluated, uncertainty due to randomly sampling permutations disappears, although discreteness remains. With random sampling, additional draws improve numerical precision but do not correct an invalid permutation scheme.

Calculate a hypothetical lower resolution

Suppose B = 999 random permutations are used and b = 0 meet the specified extremeness criterion. Under the common adjusted calculation (b + 1)/(B + 1), the estimate is 1/1,000 = 0.001, not zero. If b = 4, the result is 5/1,000 = 0.005.

These values show the reporting resolution of that calculation. They do not guarantee that the true permutation-tail probability equals the estimate. A result near a decision threshold can change with another random stream, so plan computational precision rather than stopping as soon as a favorable value appears.

Report enough to reproduce the result

Keep the seed or stream specification, software version, draw count, statistic and p-value formula. If adaptive stopping is used, describe a valid prespecified procedure; repeated informal checking can change the interpretation of the reported result.

Use the existing Monte Carlo precision guide to distinguish simulation error from scientific uncertainty. For a very small tail probability, a modest number of draws may provide poor relative precision even when the printed value has several decimal places. Finish with a statement such as a randomized permutation estimate under the specified scheme, accompanied by its computational settings, rather than a claim that no chance explanation is possible because a display rounded to zero.

Source verification: 6 November 2026. Examples are hypothetical.