Pitfalls When Using Area Under the Curve to Evaluate Item Content for Early Screening Tests for Autism

Evaluations of early screening tests for autism commonly rely on receiver operating characteristic (ROC) analysis and comparisons of area under the curve (AUC). Whether AUC differs significantly from chance or between test items is not always assessed. Two recent and independent evaluations of the Brief Autism Detection in Early Childhood (BADEC) constructed a short-form by selecting the five items with the highest AUC values, leading to inconsistencies regarding appropriate item content (Nah et al., 2018; Nevill et al., 2019). Using significance testing to compare AUC values for each test item from each dataset, we demonstrate which items justify inclusion in the BADEC, which items can be ruled out, and highlight key factors influencing AUC significance testing outcomes.