Many companies are scrambling to develop medical machine-learning tech — will the FDA stand in their way?
Machine learning has the potential to transform healthcare, leading to faster and more accurate diagnoses, as well as more effective treatment programs. While it’s already being employed by medical organizations in a number of different areas, the pace of innovation is increasing exponentially, meaning that we’re probably not far from widespread adoption.
Of course, when it comes to implementing this technology (any new technology, really) in a clinical setting, there’s always the concern of increased risk — in this case, that a technician will misinterpret the software’s findings, or that the software itself will present false negatives or positives. Of course, it’s the FDA’s job to ensure that these technologies are safe and effective. Here’s a brief history of its relationship with machine learning, as well as some insight into its approach to evaluating them in the future.
The FDA has reviewed machine learning software since 1998, when its Division of Radiological Health began reviewing computer-aided diagnostic tools. This technology identifies abnormalities and highlight them, indicating that the radiologist should take a closer look. Though the software presents little to no risk in theory, the FDA categorized it as a class III product, a ranking with the highest level of oversight and longest approval process. This is, in part, because the agency feared radiologists would rely on the program’s findings more heavily than recommended, leading to inaccurate diagnoses.
In 2012, the FDA published a series of guidance documents that discussed its approach to reviewing computer-assisted detection (CAD) devices and software. They differentiated between CADe, which highlights areas of interest but has no diagnostic capabilities, and CADx, which takes the extra step of indicating the likelihood that a lesion is cancerous. At the time, they argued that CADx software should be regulated more stringently because of the increased risk it poses, but they have since downgraded it to class II. This means manufacturers need only prove that their software is comparable to other products on the market.
There is, however, a significant burden of proof that machine learning software has to overcome before it can be widely used in a medical setting. First, clinical trials for a given product must establish ground truth — or directly observable evidence — for what an image represents. If researchers can prove the presence or absence of disease in an image, confirmed through other diagnostic techniques, then ground truth exists. The next stage is to run comparison trials, where some radiologists use the software and others do not, to determine effectiveness. The FDA is also concerned with the development of the software itself, the validity of the data sets used, and the intended use compared to the function of the product.
One of the most tantalizing promises of machine learning is that computers will eventually be able to adapt to new information they receive in the field. But this promise poses a difficult challenge, both for manufacturers and for the FDA. As regulation stands, adaptive software would need new clearances or approvals for any change to the device or algorithm. Since this is highly impractical, the FDA must settle the question of when those new approvals would be required. Adaptive systems may also require a variety of test data sets to avoid “training to the test,” so to speak. These questions must be answered before machine learning software can become widespread.
There are four major takeaways for healthcare technology companies hoping to bring machine learning software to market. First, companies have to consider intended use and how far to push boundaries of approval with their technology, lest they risk denial. Second, the adaptive component of machine learning means that a new classification system is almost inevitable, and that new technologies will be heavily scrutinized. Third, establishing ground truth is critical to approval, and may require more creative clinical trial design. Last, the FDA will apply its technical standards from radiology software to other areas, employing radiological health experts to consult on machine learning technology.
Ultimately, the FDA recognizes the value of machine learning and its significant potential for improving healthcare. The agency is working to simplify the pathway to approval for many of these technologies, and has indicated from past behavior that — health and safety considerations aside — it will not stand in the way of progress, especially progress that saves lives.