AI Healthcare Firm Claims Its Chatbot Can Beat Doctors at Medical Exams

On Wednesday night, doctors at London’s Royal College of Physicians were subjected to the world's first demonstration of an artificial intelligence (AI) robot performing a clinical test. The point of the event was to show how well the chatbot, engineered by digital medicare startup Babylon Health, would perform at the MRCGP exam, the Royal College of General Practitioners final test for trainee doctors.

In the last five years, general practitioners have averaged a 72% score at the exam, declared director at Babylon Health Dr. Mobasher Butt before announcing to the audience his bot's score. “It got 82%," said the medical expert as people began to clap.

Reducing healthcare burdens

“Tonight’s results clearly illustrate how AI-augmented health services can reduce the burden on healthcare systems around the world. Our mission is to put accessible and affordable health services into the hands of every person on Earth,” said Dr Ali Parsa, Babylon Health's founder and CEO, in a statement posted after the event.

The release revealed that Babylon Health's team had also collaborated with the Royal College of Physicians' Chief of General Primary Care, Division of Primary Care and Population Health, Stanford University Dr Megan Mahoney and Chief Population Health Officer, Yale New Haven Health Dr Arnold DoRosario to further evaluate the AI's abilities. The experiments saw the chatbot tested against seven experienced primary care doctors.

The study yielded some impressive results. For starters, the AI bot scored 80% for overall accuracy, compared to 64- 94% for the medical experts.

The bot further also performed at 98% accuracy on conditions seen most frequently in primary care medicine, compared to a 52-99% range achieved by the physicians. Most crucially perhaps, the safety of the AI was reported by the researchers to be at 97%. 

A healthcare landscape in danger

To explain his bot's purpose, Parsa shared some troubling statistics on today's healthcare conditions revealing that the World Health Organisation estimates a shortage of over 5 million doctors worldwide. More than half the world’s population does not have access to "basic healthcare services" while primary care continues to become more "unaffordable and inconvenient" with increasingly discouraging waiting times.

"Babylon’s latest artificial intelligence capabilities show that it is possible for anyone, irrespective of their geography, wealth or circumstances, to have free access to health advice that is on-par with top-rated practicing clinicians," said Parsa. 

[see-also]

The firm has published their study's results in a paper available on their website. The team also plans to make their research available through e-print service ArXiv.com.

In the paper, the authors stipulate that their new AI system "may hold the promise of reduced costs and improved access to healthcare worldwide" but add that this would entail acquiring "greater levels of confidence from the medical community and the wider public." The researchers also conclude that "studies using larger, real-world cohorts will be required to demonstrate the relative performance of these systems to human doctors."