A commercially available AI tool identified 39% more intracranial aneurysms than radiologists working alone when deployed in “shadow mode” at a large U.S. integrated health system, according to a recent study.
A group from Northwell Health in New Hyde Park, New York, prospectively evaluated an AI device on 3,856 consecutive CT angiography (CTA) exams across inpatient, emergency, and outpatient settings, and while performance varied among settings, findings support its use as a collaborative diagnostic tool.
“The AI tool demonstrated favorable operational performance, supporting clinical deployment in real-world practice,” wrote lead author Shlomit Goldberg-Stein, MD, and colleagues. The study was published September 15 in the Journal of the American College of Radiology.
Intracranial aneurysms occur in up to 4% of the general population and carry a fatality rate of up to 50% upon rupture, the group wrote. Brain CTA is the standard detection modality, but identification can be challenging even for experienced radiologists, particularly under reader fatigue or with small lesions arising from plaque-laden vasculature, they added. Whether AI can provide sufficient incremental yield to justify deployment in clinical workflows has remains uncertain, they noted.
To evaluate the operational performance of a tool (Aidoc, Tel Aviv, Israel) designed for the purpose, the researchers conducted a prospective shadow-mode study (with the AI running parallel alongside routine clinical reads) of 3,856 consecutive brain CTAs between November 7 to December 19, 2023. Radiologists interpreted examinations independently, blinded to AI results. Discordant cases were adjudicated by neuroradiology experts. Key performance metrics were the relative enhanced detection rate (rEDR), gain-to-pain ratio (GPR), and number-needed-to-examine (NNE).
Key findings included the following:
-
The AI tool identified 55 additional aneurysms missed by radiologists, achieving a 39% rEDR with a favorable GPR of 1.20.
-
AI operational value varied substantially by clinical setting, with inpatients demonstrating the strongest benefit (GPR 2.57), emergency cases were balanced (GPR 1.00), and the outpatient setting was unfavorable (GPR 0.67).
-
AI preferentially detected smaller aneurysms ( 5 mm) and those in the vertebrobasilar circulation.
“These findings underscore that AI and radiologists exhibit complementary detection profiles, supporting a collaborative deployment model,” the authors wrote.
The authors suggested that health systems may reasonably consider differential deployment strategies, prioritizing standard AI implementation in inpatient and emergency settings where value is demonstrated, while restricting use in outpatient workflows where false-alert burden may outweigh incremental yield.
“We suggest future prospective studies focus on clinical outcomes and the cost-effectiveness of AI-augmented aneurysm detection, including downstream imaging and clinical follow-up, in particular for the smaller aneurysms uniquely detected by AI,” the authors concluded.
Read the full study here.