Data scientists have developed a new taxonomy to clear up confusion about the use cases of more than 1,000 U.S. Food and Drug Administration (FDA)-authorized AI/ML-enabled medical devices, a majority of which were built for radiology.
The database and online tool classify the AI/ML medical devices and define them along different factors such as clinical function, AI function, and data type used as the core input to perform the AI function, i.e., images or electronic health record, according to corresponding author Bill Lotter, PhD, of the Dana-Farber Cancer Institute and Harvard Medical School in Boston.
“For the whole regulatory community and device manufacturers, it is hard to formulate concrete questions and suggestions unless you know where we are in the field,” Lotter told AuntMinnie.
A hot topic has been around validation and what is the appropriate level of validation and performance data a device manufacturer should submit in order to be able to get FDA clearance, he added.
“My hope is that this database can help us ask this question in a more concrete way, rather than thinking about how we should validate AI in general [but] for any given application what is the appropriate level of evidence,” Lotter said.
Using a device-level approach, rather than an FDA-authorization level approach, the information enables a more representative view of current device distribution through December 20, 2024, according to Lotter. His team’s report was published July 1 in npj Digital Medicine.
For devices based on imaging, radiology comprised the majority (88.2%), followed by neurology (2.9%), and hematology (1.9%), according to their analysis. Lotter said the work is important for radiologists and radiology practice administrators to get a sense of what devices actually do, suggesting that publicly available information through the FDA may not be sufficient.
To that end, Lotter’s team provided several key definitions relative to FDA-authorized radiology AI/ML products. For example, their definition of triage devices is that they output a binary prediction for a given exam or time series that serves as a notification for prioritized review by a clinician.
Quantification and feature localization is defined as a joint category because the functions are often performed in tandem, such as segmenting an anatomical structure on a medical image and then quantifying its volume, according to the paper.
The group determined that when the AI assists with data generation and/or the data analysis process, generation fell into three categories: image enhancement, acquisition guidance, and synthetic data generation. They noted, for example, that an ultrasound scanner that acquires images without using AI but has a built-in AI component to analyze the resulting images would be classified as analysis.
In another example, devices indicated for diagnosis do not explicitly identify suspicious regions but instead output a score or category across the input data that indicates the likelihood of a specific diagnosis, according to the team.
Predictive devices generate a score or category to indicate the future risk of an event or disease rather than the current risk, they wrote.
To create the database, the team started with the FDA’s published list of AI/ML-enabled devices authorized up to September 27, 2024. First, they reviewed all the unique product codes across the devices, including reviewing the description and example authorizations for each code, as well as public authorization summaries of the FDA’s 510(k), De Novo, and premarket approval (PMA) databases.
For each product code, the team determined whether the taxonomy values could potentially vary across devices within the product code, or whether all devices would clearly have the same values. The QFM product code, for example, has a description of “radiological computer-aided triage and notification software,” which would correspond to the image data type and an AI function subclass of triage for all devices with this code, the authors noted.
Bill Lotter, PhD, of the Dana-Farber Cancer Institute and Harvard Medical School, explains high-level findings of his team’s analysis of 1,016 AI/ML-enabled medical devices cleared by the FDA through December 20, 2024.
The team highlighted that a given device can have multiple authorizations over time for updated versions. Therefore, they grouped the 1,016 authorizations into a list of unique devices (736) within the larger context of total authorizations. Specifically, Lotter’s team found the following:
In terms of clinical function, 84.1% of current devices assist in patient assessment, such as diagnosis or monitoring, rather than intervention, meaning surgery or radiotherapy planning.
In terms of AI function, the majority (85.6%) of devices fell under analysis, rather than generation. Further, across the analysis devices, 65% of analysis devices were subclassified as performing quantification/feature localization. Triage was the next most common analysis subclass (12.9%). Detection, diagnosis, and combination detection/diagnosis devices each comprised less than 8%.
There are currently only three EHR-based devices. Currently they use tabular data such as treatment information and vital measurements as input.
The paper also connects to a website featuring an interactive flow diagram that parses data queries and provides downloadable reports.
How to explore the taxonomy using the online interactive flow diagram and database the team developed.
Read the full paper here.