Data transparency appears lacking for some AI- and machine learning (ML)-enabled medical devices cleared for Alzheimer’s disease and related dementia, according to research presented at the Alzheimer’s Association International Conference (AAIC) in Toronto.
Lack of transparency raises concerns about algorithmic bias impacting care planning for patients with Alzheimer’s disease and related dementia (ADRD), said presenter Krista Chen from the Johns Hopkins University School of Medicine. The findings were published July 30 in JAMA.
The investigators assessed U.S. Food and Drug Administration (FDA)-authorized devices for characteristics and data quality of evidence, evaluating availability and representativeness (i.e., disease type and demographics) of data supporting authorization.
First, the team’s data search identified 24 AI- and ML-based devices for ADRD authorized by the FDA between January 2015 and December 2024. Of them, 22 (91.7%) were cleared through the 510(k) pathway and two (8.3%) through de novo classification, both low-risk pathways with lower evidence standards, Chen said during her AAIC talk.
Using FDA summaries and peer-reviewed articles as reference documents, Chen and colleagues looked for available training data and validation data. The group found the following:
- For 12 devices, they found no information on training or validation datasets available in either FDA summaries or peer-reviewed articles.
- FDA-approval summaries provided training data for 10 devices (41.7%), with peer-reviewed articles offering training data for five (20.8%). Data regarding disease type and age varied, with race and ethnicity rarely disclosed.
- Very few FDA summaries provided external validation data — only two (8.3%) — but peer-reviewed articles provided external validation data for 10 (41.7%). Again, patient disease status, age, and sex varied in validation data, with race and ethnicity rarely reported.
The researchers also found no justification for 23 devices with incomplete reporting across the domains of disease status, age, sex, race, and ethnicity.
Lack of data raises uncertainty about real-world generalizability and clinical accuracy in intended populations, according to Chen and colleagues. The devices were primarily diagnostic, Chen noted at AAIC.
|
Indications and number of AI- and machine learning-based medical devices for Alzheimer’s disease and related dementias* |
|
| Volumetric quantification of brain structures using MRI / including for hippocampal quantification only |
20/3 |
| Functional testing for memory, learning, or visuospatial awareness |
3 |
| Quantification of amyloid deposition using PET |
2 |
| *Cleared for two indications |
1 |
Most (21) of the FDA-cleared AI- and machine learning-based medical devices for ADRD were reviewed by radiology panels. Three were reviewed by neurology panels.
“Data transparency — particularly regarding demographic representativeness of training and validation datasets — is essential to understanding performance variability and ensuring appropriate application in intended populations,” Chen and colleagues wrote.
The group also highlighted FDA guiding principles for good machine-learning practice for medical device development, as well as updated draft guidance in 2025, both of which set forth recommendations around demographics with AI and machine-learning medical devices.
“To this day, it is unknown whether these practices have actually been enforced,” Chen said during her talk.
Enforcing guidelines for data transparency and device labeling would help regulators identify potential biases and promote safe and effective AI and machine-learning deployment in dementia, the investigators noted.
“While truly representative datasets may be an impractical goal, transparency and better understanding of what goes into datasets is essential,” Chen said at the conference.
Read the complete paper here.