Evaluate Screening Automation Before Letting It Exclude Records

Specify the automation’s role

A system that orders records for human screening makes a different decision from one that removes records without review. State which role is proposed, what input it receives and what its output controls. Saving time is not sufficient evidence that automatic exclusions are appropriate for the current review.

Cochrane’s study-selection guidance discusses software support and automation within a structured selection process. Evaluate the actual workflow, including human checking, rather than treating a product name as a validated screening method.

Build a defensible evaluation set

Use records whose eligibility has been independently adjudicated under the current criteria. Keep the evaluation records separate from records used to tune thresholds or prompts. Include difficult terminology, uncommon designs and records with incomplete abstracts. A convenient set containing only obvious exclusions will overstate practical performance.

Record the software version, decision threshold, prompt if relevant, and handling of missing metadata. Preserve predictions before revealing adjudicated labels. If the system changes during the review, determine whether the evaluation still applies and retain the earlier results.

Inspect a hypothetical confusion table

Suppose 200 evaluation records contain 20 eligible records. A proposed exclusion rule would discard 150 records, including two eligible records. It retains 18 of the 20 eligible records, so sensitivity on this set is 18/20 = 90%. Its apparent workload reduction is 150/200 = 75%, but that reduction comes with two missed eligible records.

Neither percentage establishes acceptable performance. The eligible sample is small, uncertainty matters, and the evaluation set may differ from future search results. Investigate the two errors: a recurring terminology failure may be more informative than an aggregate accuracy score dominated by ineligible records.

Define safeguards and stop rules

Specify who reviews uncertain records, how exclusions are audited and what error triggers suspension or retraining. Do not stop screening merely because the most recent batch contains no inclusions unless the stopping method has an appropriate justification for this task.

For assisted ranking, retain the full record set and ensure that prioritization does not quietly become exclusion through fatigue or an arbitrary deadline. Report the role of automation, evaluation design and human responsibilities transparently. The deliverable is a tested decision process with identifiable limits, not a claim that a high overall accuracy percentage certifies a complete evidence search.

Source verification: 6 November 2026. Examples are hypothetical.