Sunday, November 30 | 11:20 a.m.-11:30 a.m. | SSCA01-6 | Room E353C
If this scientific presentation is any indication, AI software updates should undergo local validation and performance monitoring following deployment.
Led by Syed Muhammad Awais Bukhari, MD, of Cleveland Medical Center, researchers compared two versions (v1.3 and v1.4) of a U.S. Food and Drug Administration (FDA)-cleared AI algorithm for coronary artery calcium (CAC) scoring and stent detection.
Internal improvements with the updated version 1.4 included adding training data and additional model parameters, improving CAC segmentation, and enhancing stent detection, according to Bukhari and colleagues.
To check accuracy and radiologists’ preferences after the software update, researchers retrospectively applied v1.3 and v1.4 to 110 consecutive low-dose CT (LDCT) scans.
The group found that CAC v1.4 demonstrated improved coronary artery stent detection; the updated version correctly identified eight of 10 known to be in the cohort, they found. In contrast, version 1.3 identified none. On the other hand, the updated v1.4 consistently underestimated CAC scores relative to the earlier v1.3, according to the group.
The group highlighted that in patients without stents, the updated software produced a mean CAC score significantly lower than the earlier v1.3 (288.5 compared with 452.7).
This prompted concerns about cardiovascular risk misclassification, they noted. For example, 18 cases shifted from moderate to minimal (1-99), and one shifted from severe to minimal.
Therefore, radiologist preference and statistical evidence favored version v1.3 in a majority (58.2%) of cases, according to the group. Radiologists disagreed with both outputs in only six of the 110 cases.
This session references the FDA’s Predetermined Change Control Plan (PCCP) framework for AI-enabled medical devices. “These findings highlight the importance of local validation and ongoing performance monitoring before adopting AI updates under the PCCP framework,” the group said.
Stop by with your questions.