Artificial intelligence (“AI”) will likely drive many complex medical devices in the near future. But as with all things, AI can sometimes fail. Companies relying on AI to elevate their medical devices above the competition should be mindful of four common AI failure modes: AI functional errors, software rot, unexplained programming glitches, and the ever-present human factor.
Most AI driven systems currently rely on computer vision, time series analysis, speech recognition, and/or natural language processing to carry out their intended tasks.
Computer vision is where the AI analyzes images and video. This capability has several important applications in the healthcare space. For example, it can be used in patient diagnosis. AI can help more accurately interpret things like radiology images. It can also classify images into diagnostic categories, potentially lessening the chances of misdiagnosis. AI can also perform image analysis during robot-assisted surgeries. It can identify objects in video data and provide real-time guidance to surgeons during the operation. Such a capability is important because it may help reduce human error and lead to more optimal patient outcomes.
Time series analysis is where AI is used to analyze temporal data and detect anomalies or make predictions. In other words, the AI detects and interprets certain things in relation to time. So for example, AI can analyze electrocardiogram data from a wearable device and detect things like arrhythmias.
Speech recognition can be broadly divided into two main categories. The first is where AI is used to analyze spoken language and detect certain medical conditions or diseases. For example, AI can listen to speech and by analyzing things like pitch, tonal quality, and rhythm, detect physical symptoms like throat inflammation, or even ailments like PTSD, depression, Alzheimer’s, and Parkinson’s. The second category is plain old speech recognition, which is used for issuing voice commands and systems like virtual assistants.
Finally, we have natural language processing. This differs from speech recognition in that the AI not only identifies speech, but it actually extracts meaning from natural human language. This capability can be used to extract data from electronic health records. It can also be used in systems like chat bots to provide information to patients and assist with functions like scheduling and billing.
Unfortunately, AI driven medical devices can fail when one or more of these capabilities don’t function as intended. For example, a benign tumor can be misdiagnosed as a malignant tumor, or vice versa, because the AI misinterpreted an image. And if the AI is being used in a surgical robot, object misidentification can result in the AI providing bad surgical guidance. This can lead to surgical error and patient physical injury. A patient may also be misdiagnosed due to speech recognition failures based on variations in speaker accents or dialects. Such misdiagnosis may result in the AI recommending the wrong drug to a patient, or even inaccurately predicting adverse drug events.
Software rot is another commonly overlooked but important factor worth considering. This is the slow deterioration of software quality over time, which eventually leads to it becoming faulty, unresponsive, unreliable, or completely unusable. Basically, think of how your smartphone slows down over time and starts malfunctioning. Several factors though to be responsible for this phenomenon include changes to the software’s operational environment, degradation of compatibility between parts of the software itself, and the appearance of bugs in unused or rarely used code. Software rot can lead to many of the AI failures described above.
Unexplained programming glitches can also pose a significant risk. Modern AI runs on algorithms containing billions of lines of code. Certain interactions in that code can result in unexpected and baffling results. A good example of such glitches can be seen in modern video games, where pre-programmed AI routines sometimes take strange turns, making characters act in wildly unpredictable ways. Similar issues can occur within AI routines programmed into medical devices, leading to many of the AI failures described above.
Finally, the human factor should never be overlooked. The way humans impact AI performance can be broadly split up into two categories – human error and intentional improper use. A good example of human error includes data entry errors resulting in inaccurate prediction by properly functioning AI systems. In other words, you can have a situation where an AI predicts that a patient is unlikely to have a certain disease based on something like a typo in lab results. Another good example is where a physician recommends the use of an AI driven surgical system for a patient who is not a good candidate. In fact, this was one of the allegations in Taylor v. Intuitive Surgical where it was claimed that a surgeon used the da Vinci surgical system on an overweight patient against the manufacturer’s advice, resulting in patient injury.
Intentional improper use can also significantly impact AI performance. The FDA has made clear in recent years that medical device manufacturers bear the burden of ensuring that their products are not vulnerable to hacking. But just like any other piece of software, AI systems can be hacked by nefarious actors. Examples include hackers inserting false data into medical imaging, or taking control of insulin pumps. AI systems used to analyze electronic health records can also be hacked to locate and retrieve social security numbers, birthdates, and other sensitive information critical to identity theft. Finally, hostile foreign governments can access and manipulate AI to do their bidding. Just imagine what a foreign intelligence agency could do with a remotely accessible AI controlled pacemaker implanted within an important political figure.
To safeguard against these and other risks, it is important for companies that manufacture and sell AI driven medical devices to design their products in ways that account for these and other common failure modes. Such design elements should include—among other things—the ability to detect anomalies and improper use, patch software in a timely manner, and potentially prevent the device from operating without the latest software update. Companies should also partner with experienced legal counsel to monitor trends and developments in this space, and chart effective legal strategies to eliminate or mitigate liability exposure in the event that something goes awry.