Can ChatGPT help with breast cancer screening? A study finds out

Researchers found that ChatGPT answered questions correctly about 88 percent of the time.
Jijo Malayil
Can ChatGPT screen for breast cancer
N/ACan ChatGPT screen for breast cancer


The advent of AI-backed language models like OpenAI's ChatGPT has created a keen interest among consumers, especially in the field of health, to gather medical advice. Now, researchers aim to evaluate these technologies to identify whether the information provided by artificial intelligence is reliable and accurate.

A new study conducted by a team at the University of Maryland School of Medicine (UMSOM) indicates that responses generated are often correct; however, there is a probability that it is inaccurate or even fictitious at times. The findings of the study are published in the journal Radiology.

The study was conducted by creating a set of 25 questions related to procuring advice for breast cancer screening. The team then ran each question to ChatGPT three times to analyze the responses generated. The answers were evaluated by three radiologists specialized in mammography, and they found that the responses were appropriate for 22 out of the 25 questions. 

Outdated information led to the chatbot providing an incorrect response with one question. The remaining two had inconsistent responses that varied significantly each time the same question was posed.

A significant achievement for an evolving AI-product

The researchers have expressed optimism regarding the possibilities of such a product being of help in the health sector. "We found ChatGPT answered questions correctly about 88 percent of the time, which is pretty amazing,” said Dr. Paul Yi, Assistant Professor of Diagnostic Radiology and Nuclear Medicine at UMSOM and the study's author, in a press release. The team added that the chatbot could help summarize the information into passages that were easy to comprehend for average readers. 

According to the team, the AI-bot correctly answered questions about the "symptoms of breast cancer, who is at risk, and questions on the cost, age, and frequency recommendations concerning mammograms."

Responses are not comprehensive in the area of reference

On the downside, researchers have highlighted that the information provided is not exhaustive or based on all possible references to the matter available online. “ChatGPT provided only one set of recommendations on breast cancer screening, issued from the American Cancer Society, but did not mention differing recommendations put out by the Centers for Disease Control and Prevention or the U.S. Preventative Services Task Force,” said study lead author Hana Haver, MD, a radiology resident at University of Maryland Medical Center.

What is more surprising is that, ChatGPT can often create fake journal articles or health consortiums to support its claims. "Consumers should be aware that these are new, unproven technologies and should still rely on their doctor, rather than ChatGPT, for advice,” said Dr. Yi.

The team is in the process of analyzing how ChatGPT fares for lung cancer screening recommendations and evolving ways in which ChatGPT can improve its accuracy, making it more comprehensive and "understandable to those without a high level of education."

Add Interesting Engineering to your Google News feed.
Add Interesting Engineering to your Google News feed.
message circleSHOW COMMENT (1)chevron
Job Board