Only high-performance AI programs can improve lung cancer detection, finds study

Their latest study reveals details about the type of AI programs that deliver more reliable results.

High accuracy AI vs. low accuracy AI

The study authors propose that only those AI programs with high diagnostic performance, meaning that they can detect almost all or most abnormalities in a chest X-ray and are as good as or better than radiologists in reading X-rays, should be considered for use.

They refer to such a program as a high-accuracy AI; any other AI that doesn’t meet these requirements is called a low-accuracy AI.      

“Our study suggests that AI can help radiologists, but only when the AI’s diagnostic performance meets or exceeds that of the human reader,” said Dr. Chang Min Park in a press release, one of the study authors and a professor of radiology at SNU College of Medicine.

These findings are based on an interesting experiment involving 30 human radiologists, one high-accuracy AI, one low-accuracy AI, 60 chest X-rays of patients living with lung cancer, and 60 X-rays of healthy individuals.

In the first part of the study, 20 thoracic radiologists having 10 to 18 years of experience and 10 radiology students examined all the X-rays without any AI assistance. In the second part, they used the help of both AI programs. However, the radiologists weren’t aware of the differences that exist in the diagnostic power of the two AIs. 

The researchers then compared the results of the first part of the study with the results they recorded during high and low-accuracy-assisted X-ray readings. 

When assisted by the high-accuracy AI, the radiologists reported that the per lesion sensitivity improved by 0.10 (0.53 without AI and 0.63 with AI) and specificity increased by 0.14 (0.88 without AI and 0.94 with AI). 

The abovementioned scores suggest that they could detect more abnormalities with the help of the AI program. However, no such improvements could be achieved when they took the help of low-accuracy AI. They detected almost the same number of abnormalities with and without AI.

These findings strongly suggest that AI programs that lack a high diagnostic performance won’t deliver any benefits in lung cancer diagnosis. Therefore, radiologists need to ensure that they use a high-accuracy AI.  

People are likely to trust a high-accuracy AI 

The researchers suggest that since a high-accuracy AI gives out more reliable results, such an AI may also lead “to more frequent changes in reader determinations—a concept known as susceptibility.”

“It is possible that the relatively large sample size in this study bolstered readers’ confidence in the AI’s suggestions. We think this issue of human trust in AI is what we observed in the susceptibility in this study: humans are more susceptible to AI when using high diagnostic performance AI,” said Park.

The definition of a high-performance or high-accuracy AI may also change as per the clinical purpose for which it is used. For instance, a high-accuracy AI ideally suited for lung nodule detection may not work well for mass screening pulmonary tuberculosis.

Therefore, the researchers recommend using a clinically appropriate AI with diagnostic performance suitable for the targeted medical screening process.

 The study was published in the journal Radiology on June 27.