Wednesday, December 3 | 9:30 a.m.-9:40 a.m. | SSER02 | Room N228
In this scientific presentation, researchers from the U.K. will share performance analysis of nine AI tools for detecting pneumothorax on chest radiographs.
Curating a dataset of 394 chest x-rays, Dr. Katrina Nash of Oxford University Hospitals in the U.K. and colleagues worked with two radiologists to establish ground truth for pneumothorax presence or absence on each chest radiograph, with disagreement arbitrated by a third radiologist.
They then retrospectively calculated sensitivity, specificity, accuracy, and receiver operating characteristic (ROC) curve metrics for each vendor tool — results of retrospective evaluation. In advance, the group highlighted accuracy ranging from 79.5% to 93.9%. The lowest specificity value hovered near 80%, they also reported. Plus, sensitivity trended lower across the entire group, with results as low as 54.3% and as high as 92.5%, according to the group.
Vendors were able to provide variable threshold scores, they noted.
While multiple AI algorithms demonstrated high accuracy in identifying pneumothorax on chest x-ray, significant variation in accuracy persisted across the group, the results indicated. AI performance differences have important clinical implications, Nash and colleagues noted.
Where will these AI tools likely result in false negatives in clinical practice and, therefore, increase the risk of missed pneumothorax on chest radiographs? What steps could improve performance?
This session — designated for the RSNA 2025 Trainee Research Prize (Resident) — confirms that comparative evaluations of AI algorithms across all relevant pathologies using large datasets representative of the intended population are critically needed to ensure clinical safety prior to deployment.
Join to discuss.