Compare Simulation Bias and RMSE Without Treating Them as Interchangeable

Store the true value with every scenario

Bias compares the average estimate across simulation replicates with the true target value used in the data-generating mechanism. Root mean squared error combines the sizes of estimation errors through squared deviations. A method can have low bias but substantial variability, or some bias with smaller overall error.

The rsimsum package documentation includes bias and mean squared error among its performance summaries. Before computing either, confirm that every method estimates the same target or explicitly label different estimands.

Define the denominator and scale

Record which replicates contribute valid estimates and whether the summary excludes failures. Compute errors on the scale relevant to the scientific question. Bias on a log scale is not automatically the logarithm of bias on the original ratio scale.

Relative bias divides by the true value and becomes unstable or undefined near zero. Use absolute bias or another meaningful scaling when a zero truth is part of the scenario grid. Do not hide this issue by replacing a zero denominator with an arbitrary small constant.

Calculate a hypothetical comparison

Suppose the true value is 10 and four illustrative estimates are 8, 9, 11 and 12. Their average is 10, so empirical bias is zero. Their errors are −2, −1, 1 and 2; squared errors average (4+1+1+4)/4 = 2.5, giving RMSE √2.5 = 1.5811.

A second method always returning 11 has bias 1 and RMSE 1 in this tiny illustration. It has more systematic error but lower squared error. This example explains the distinction; four fabricated estimates are not a simulation study capable of establishing general superiority.

Report uncertainty and context

Use enough independent replicates and report Monte Carlo uncertainty for performance summaries. Inspect the distribution of estimates, not only their average, because occasional catastrophic errors can dominate RMSE. Such behavior may be scientifically important rather than a nuisance to trim away.

Present bias, variability, RMSE and failure rates together when they answer the methodological question. Keep scenario-specific truths visible and avoid averaging across incompatible scales. The final comparison should make the tradeoff between systematic error and variability understandable, rather than ranking methods by a single number without explaining what that number rewards.

Source verification: 6 November 2026. Examples are hypothetical.