Claim: AI is “Much Better Informed Than Any Doctor in the Country”

Verdict: False and misleading

“You may have a medical record that is a thousand pages long. You have six minutes with a doctor today. He is not gonna be able to review it, but the AI can and it can distill it. It can give you a second opinion that is much better informed than any doctor in the country.”

– RFK Jr.

VP Vance speaks at MAHA Summit with HHS Sec. Kennedy

Speaking at the MAHA Summit in Washington, last week RFK Jr. claimed that AI is “much better informed than any doctor in the country.” While RFK Jr. did mention some undeniable advantages – quickly reading long medical histories, for instance – how does the overall claim stack up?

What Do Doctors Think?

Major medical organisations, including the American Medical Association, pushed back RFK Jr.’s statement, stressing the value of physicians’ years of training and experience. The statement emphasises the benefit of using the technology responsibly, as a tool to help doctors, guided by the “humanity of clinical practice.”

What Does the Evidence Say?

There are some studies showing a benefit of AI systems over human doctors, including one oft-cited meta-analysis result on empathy. This study typifies an issue with much of the research in the area. Of 15 studies included in the study, 13 showed higher empathy ratings for AI, compared to human doctors. However, the interactions studied were only through text. This severely limits applicability to real-world situations, where most medical evaluations are face-to-face.

Another meta-analysis compared diagnostic capabilities of AI with human doctors, based on 83 studies. The results showed that AI performed roughly as well as physicians overall, or non-expert physicians. However, it was worse than expert physicians.

A controlled study touched on the “second opinion” aspect of RFK Jr.’s claim, looking at whether LLMs can help members of the public choose accurate diagnoses and treatments. Generally, this study found that the LLM did better alone than it did assisting a human. When assisting non-experts – who may not communicate clearly – it does no better than participants findings their own online sources. 

The Final Verdict

LLMs may play many valid roles in healthcare both now and in the future, but they certainly are not better informed than any doctor in the US. Nor is it a particularly useful option for non-experts seeking medical advice.