Count Method Failures as Simulation Results

Define failure before the main run

A method can fail by throwing an error, failing to converge, returning a nonfinite estimate or producing an invalid uncertainty measure. Specify these categories before reviewing comparative performance. Keep warnings separate from hard failures when a warning requires further inspection rather than automatic rejection.

The rsimsum documentation notes handling of missing estimates and boundary exclusions. That makes an explicit failure ledger essential: a summary based only on available outputs may describe a selected subset of easier datasets.

Preserve each attempted fit

Create one record per scenario, replicate and method, even when fitting fails. Store status, error category, runtime and any available diagnostic values. Retain the dataset’s reproducible stream identifier so failures can be investigated without guessing which random sample caused them.

Use a prespecified retry policy if different starts or tolerances are allowed. Record both the original failure and the retry result. A method that needs rescue on half its datasets should not be reported as uniformly successful merely because a later fallback returned numbers.

Reconcile a hypothetical comparison

Suppose methods A and B are each attempted on 1,000 common simulated datasets. A returns valid estimates on 990 and B on 800. Their failure rates are 1% and 20%. A bias summary for B based on 800 successful fits is conditional on that success subset.

If B fails mainly in datasets with rare outcomes, its successful-case bias may look small while concealing poor applicability in exactly the difficult setting under study. Compare performance on common successful replicates as a clearly labeled secondary analysis, but retain the overall failure rates and explain the selection.

Investigate causes rather than suppressing them

Review representative failures by scenario. Distinguish implementation bugs from inherent estimator limitations and numerical configuration problems. Fixing a bug requires rerunning affected outputs under a new code version, not deleting inconvenient failures from the results table.

Report attempted, successful and failed counts for every method-scenario combination. State the denominator used for each performance measure and any fallback method included in the definition of success. A simulation comparison is useful only when readers can see both the quality of returned estimates and the conditions under which a method could not provide a trustworthy result.

Source verification: 6 November 2026. Examples are hypothetical.