New AI Is More Than Twice as Good at Diagnosing Diseases as Doctors

How do you make corn when you’ve eaten all the seed?

If I needed someone to build me a complicated application, I would choose the software developer with 20 years of experience before LLM's were capable of helping software developers that uses LLM's to assist over the software developer with 1 year of experience that never learned to code and just uses LLM's for everything but is clueless as to how any of it works.

Same thing with doctors.

1. This a claim from Microsoft not a peer reviewed study on pubmed. So it's a marketing title nothing more

2. Even some ideal frontier model is only as accurate as the inputs given for diagnosis, and will typically bias towards said inputs.

3. Multi-modal can handle images but are terrible at image classification unless using dedicated models for each pathology.

4. The very large majority of traning corpus with these companies is in English. There is zero guarantee that said mythical performance is preserved across languages.

5. The very thought of companies with the moral compass of most AI companies accessing and training on your medical records should make anyone pause.

I'm not sure what AI companies have to do with it. Open weight models are available and work great, literally today kimi K3.1 pro was announced and goes toe to toe with closed models. You are tying AI usage to bigAI and that is unfounded.

AI is going to be both a boon and a bane for doctors. Its "good enough" behavior will lead to complacency like it does any place where the automated system is "good enough" but the system still wants human validation. This will without a doubt result in some preventable tragedies that will end with more work and check lists (and probably not enough accountability). But there are other places where it will absolutely free up mental energy to focus more on the patient.

I was talking to my own doctor and their experience with a new note taking / transcription AI app they've been using, having worked on medical software before. One of the problems I noticed when designing software then was that software likes its categorized fields, with data entered just the right way and in just the right order. But medical work is almost never that neat and tidy. Patients and other medical professionals give you information in whatever order things come to their mind in, or in whatever order their own system and paperwork displays it. A simple prescription consisting of a patient's name, DOB, medication, medication form, dispensing quantity, refills and dosing instructions might be given to a provider in any order of those fields and no matter what order your software chooses, it won't be the same order as someone else giving it to you. Most providers that used the software I worked on resorted to hand writing everything down and filling it all in later, or having to ask for repetition over and over as they clicked through various fields trying to remember everything before it flicked out of their short term memory or the other person moved on. And as more and more systems and regulations require everything to be entered into electronic records, this has become more of a problem. To the point where you go to some appointments and you often don't even see your doctor's face because they're stuck behind a terminal nodding along as they try to capture everything you say into the specific fields in the system. AI tooling has the potential (and at least from my own doctor's experience, is already showing that potential) to free up the mental effort spent on getting the right data into the right fields. AI can do a great job of understanding that a free text field that says:

    Fucitol tab 25mg qid
    John Smith 01/01/75
    #30 3 months

and a free text field that says:

    Fucitol 25mg tablet
    30 tabs, 4x per day
    3 refills
    Smith, John 1975-01-01

say the same things, and get those things into the right fields in the record for you. They still need to go back and check it (and see the previous caveat about "good enough" systems) but by freeing the doctor from the mental load of intentionally categorizing all of the data they're hearing while simultaneously trying to reason through what all that data means and enter it at the same time they have more mental room to actually pay attention to the things the patient is saying, the things they're doing and the things the doctor is seeing.

And one thing I hope is that it makes switching doctors if you need to safer and more effective. The sheer volume of information a new patient can bring with them if they have all their medical records is massive and sorting through it all is likely something we'll never give our providers enough time and money to do. Things will get missed, or doctors will defer (consciously or not) to the diagnosis already listed, even if a more thorough review might reveal the diagnosis was wrong. An AI system designed to review an influx of data with an adversarial eye, or at least an eye towards looking for missed possibilities or potential reasons for the patient's dissatisfaction with the previous provider could go a long way towards helping end the sorts of "I went through 10 doctors all of which told me it was all in my head/I should try losing weight/I can't be experiencing this issue at my age, and then finally the last doctor had just finished reading a paper about how XYZ can cause these symptoms and now I'm finally getting treatment" stories anyone with friends or family with chronic health problems probably has heard more than once.

Averages are not much use if you are the patient whose diagnosis was missed

Because we have limited resources, time and every intervention can cause more harm. For example, you don't want to get an MRI every time you get a headache - it would be counterproductive to you, drain resources from a system which needs to handle more patients, and regularly make you visit an environment with potential infectious diseases. Where we have a quick and simple empirical test, we often apply it and the available options are growing - for example we have quick tests for a number of respiratory diseases, which wasn't available earlier.

The problem is that insurance won't authorize unless the doctor has a good reason to suspect a rare decease with expensive tests. Even then they will verify that the doctor has followed protocol and gone thru all the cheap tests first.

Not exactly what?! Say that you don't agree and then produce an argument as to why!

Mammograms are necessary so are tests when the blood/urine/fecal tests indicate abnormalities. How is liver or pancreas, kidney, stomach cancer detected? magic? Certain tests are unavoidable because doctors don't have xray eyes.

Apparently there is going to be Multi-Cancer Early Detection Tests if they get FDA approval. Not accurate enough yet but probably better than nothing.