Measuring the Efficiency of Automation-Aided Performance in a Simulated Baggage Screening Task

Melanie M. Boskemper, Megan L. Bartlett, Jason S. McCarley

Research output: Contribution to journalArticlepeer-review

Abstract

Objective: The present study replicated and extended prior findings of suboptimal automation use in a signal detection task, benchmarking automation-aided performance to the predictions of several statistical models of collaborative decision making. Background: Though automated decision aids can assist human operators to perform complex tasks, operators often use the aids suboptimally, achieving performance lower than statistically ideal. Method: Participants performed a simulated security screening task requiring them to judge whether a target (a knife) was present or absent in a series of colored X-ray images of passenger baggage. They completed the task both with and without assistance from a 93%-reliable automated decision aid that provided a binary text diagnosis. A series of three experiments varied task characteristics including the timing of the aid’s judgment relative to the raw stimuli, target certainty, and target prevalence. Results and Conclusion: Automation-aided performance fell closest to the predictions of the most suboptimal model under consideration, one which assumes the participant defers to the aid’s diagnosis with a probability of 50%. Performance was similar across experiments. Application: Results suggest that human operators’ performance when undertaking a naturalistic search task falls far short of optimal and far lower than prior findings using an abstract signal detection task.

Original languageEnglish
Pages (from-to)1-17
Number of pages17
JournalHuman Factors
Early online date2021
DOIs
Publication statusE-pub ahead of print - 2021

Keywords

  • decision-making strategies
  • human–automation interaction
  • naturalistic visual search
  • signal detection theory

Fingerprint Dive into the research topics of 'Measuring the Efficiency of Automation-Aided Performance in a Simulated Baggage Screening Task'. Together they form a unique fingerprint.

Cite this