AI automated radiographic scoring in rheumatoid arthritis: Shedding light on barriers to implementation through comprehensive evaluation

Objectives: Artificial intelligence (AI) has demonstrated the potential to improve efficiency and reliability of radiographic scoring in rheumatoid arthritis but lacks sufficient evidence to justify clinical use. We developed and rigorously validated a deep learning model to automate radiographic scoring against two external test sets, drawing upon state of the art reporting guidelines to clarify present barriers to implementation. 

Methods: AI algorithms were trained to predict the Sharp van der Heijde score in hands and feet using a cohort of 157 patients and 1470 radiographs. External replication was undertaken in test datasets from two hospitals (n=253 patients, 589 radiographs). Alongside standard performance metrics to measure error and agreement, we reported subgroup performance, conducted an exploratory analysis of error, and demonstrated relationships with functional outcomes. 

Results: Our AI system underperformed compared to manual scoring, with lower agreement between the AI and consensus score than between the two manual scorers. The AI system was better at ranking scores than achieving absolute agreement, with intraclass correlation coefficients ranging from 0.03 to 0.27 while Spearman's correlation coefficients were consistently higher, ranging from 0.16 to 0.55. 

Conclusions: The performance of the AI systems developed for automating radiographic scoring in RA is insufficient to justify use in research or clinical practice. Large, diverse, and thoroughly described longitudinal datasets will be indispensable in the development and rigorous evaluation of algorithms. Achieving this is key to the ongoing precise evaluation of clinical outcomes in rheumatoid arthritis to enable further improvements to patient care.