Capabilities of GPT-4 on Medical Challenge Problems
arxiv.org
arxiv.org
Now - throw a punch of clinical guidelines in a vector database and give it context and it’s 10x better than me and any doctor outside their speciality or all the mid-levels. (I.E, it’s better than cardiologist doing infectious disease - but not cardiologists doing cardiology). This because there are very niche stuff as you specialize where it’s only like 5 doctors who see it in the whole world on a consistent basis (and they don’t blog!)
I trained it on the IDSA guidelines (infectious disease) and put up a proof of concept on GalenAI.co - just as way to start talking to health systems and clinicians. it’s going to be very different world in medicine in a couple of years from now!!
Unable to ship anything, protect their margin > help the users solve problems, monopoly, locked up distribution so no one else can innovate.
Honestly, my bear case for AI in medicine is Epic picking up the phone and telling health-systems not to buy anything because they are working on something for them for free. (Which would be some note completion BS stuff, rather than actual clinical support that helps patients and cuts costs). They may be doing this already.
Of course matching that data up with sets from other locations will still involve someone in the middle gluing it all together.
As a patient I never really set eyes on the interface but there seems to be a UX nightmare afoot. Once I was unintentionally dispensed 3x the intended dose of a stimulant medication due to a “default dose” feature in the interface that my physician admitted to accidentally submitting based on
For some context, the USMLE is taken during medical school. The amount I have learned about actually practicing medicine since graduating is probably an order of magnitude more than everything I learned in medical school! I still learn stuff, all the time, and I’m not just talking about new research.
So, while impressive and clearly part of the future world, we shouldn’t get too far ahead of ourselves with the current models.
Edit: oh I should add that there are more clinically relevant exams that would be more likely to reveal d clinical usefulness, for example “board” exams. These are taken after training, usually before practice. Not knocking LLMs, just ensuring that people don’t misunderstand passing the USMLE as being clinically useful.
Congress can do a lot.
Doctors and nurses have saved me many times from some very close calls because of decades of experience, training and intuition. That is of course not to mention the friend who beat a deep brain tumor on their brain stem that everyone else told them was inoperable, and now are in medical school themselves for neurology. No LLM is going to pull that out of itself, possibly ever, and certainly not GPT-4 (no one else had ever had the surgery done before, it was novel).
Doesn't matter how much GI doctors know when all they do is scope you. Sure doc scoping is sure going to help people with atypical intolerances, IBS, and any number of modern chronic conditions for which treatments are inadequate. They have to do better!
Their output is impressive considering that they're incapable of reasoning, but it desperately needs to be sanity-checked by someone with relevant knowledge before using it for anything serious.
It does in fact create some sort of model out of those billions of dimensions, and it's able to re-use it across many different fields, too
It's not "bottom-up reasoning" so it misses a bunch of little details.
Basically AI is a search through a vast space, and most of the approaches have focused on top-down search. Reasoning bottom-up is what AlphaBeta search did (rybka and all those other chess programs, that calculated all possible combinations 10 moves ahead). Those missed some of the "top-down positional" stuff, whereas AlphaGo missed some weird-ass edge cases
https://www.engadget.com/alphago-pushed-human-go-players-to-...
But it is a sort of intelligence. Just not one you can trust when details matter. You can use it for pretty pictures for now, and essays.
Generative AI is not deep but extremely broad. It knows many subjects. And can discuss more than any specific person.
But guess who produced all that content for it and made it publicly available. Humans!
(And in cases where it wasn't publicly available, I guess they can sue the company)
I think that someone that’s gone as far to built this with GPT-n is aware of its limitations.
In one study (see reference) users were able to identify the AI about 2/3rd of the times (and identical for the human experts). So 1/3rd of the time the AI is not detected and may well be spouting nonsense.
https://www.medrxiv.org/content/10.1101/2023.01.23.23284735v...
Suppose you have 2 equations. You don't know what these equations are. However, you know that for any input, the output is the same.
Any mathematician worth his salt will tell you that given said information, those 2 equations are equal or equivalent. It doesn't actually matter what those equations are.
Equation 1 could be say a+b and Equation 2 could be (a-5) + (b-5) + 10
Or more realistically, both equations could look vastly different.
The point I'm driving home here is that true distinction reveals itself in results.
The fallacy of the philosophical zombie is that there is this supposed important distinction between "true understanding" and "fake/mimicry/whatever understanding" and yet you can't actually test for it. You can't show this supposed huge difference. A distinction that can't be tested for is not a distinction.
Now you could make the argument that these functions are close enough most of the time so it won't matter but unless you want to get really rigorous that's more of a stats / engineering perceptive not mathematics. Any more importantly that's very up for debate, especially in a high pressure situation like medicine.
Of course these models are wild and I'm quite impressed with them. I still can be worried about the damage someone who doesn't think things through could cause by assuming GPT-4 has human or super human level intelligence in a niche, high impact field.
The point I'm making is that I can quite clearly show output that demonstrates reasoning and understanding by any actual metric. That's not the problem. The problem is that when I do, the argument Quickly shifts to "it's not real understanding!". That is what is nonsensical here. It's actually nonsensical whatever domain you want to think about it in.
Either your dog fetches what you throw at it or it doesn't. The idea of "pretend fetching" as any meaningful distinction is incredibly silly and makes no sense.
If you want to tell me there's a special distinction cool but when you can't show me what that distinction is, how to test for it, the qualitative or quantitative differences then I'm going to throw your argument away because it's not a valid one. It's just an arbitrary line drawn on sand.
There is a lot of research that shows they have trouble reasoning. Folks working with LLMs and building them agree that they can't reason, yet pop-sci folks religiously insist they can reason.
This is not even a controversial topic.
https://arxiv.org/abs/2205.11502
Reminds me of this comic:
Look at section 8.2 in the paper you linked:
"8.2 Lack of planning in arithmetic/reasoning problems"
> However, it seems that the autoregressive nature of the model which forces it to solve problems in a sequential fashion sometimes poses a more profound difficulty that cannot be remedied simply by instructing the model to find a step by step solution
Nowhere in the paper is an evaluation on standard reasoning datasets (https://tptp.org) because they are much much much tougher than the simple examples that GPT-4 struggles with.
LLMs are a huge breakthrough but let us not muddy the science.
https://arxiv.org/abs/2212.09196v1
https://arxiv.org/abs/2211.09066
https://thegradient.pub/othello/
Moreover, "have trouble reasoning" and "can't reason at all" are two very different statements. Not that i particularly agree with either but you should keep that in mind.
It was pretty good, but sometimes it wouldn’t even try. It would spit out a sentence that could have been easily rewritten to start with b and end with d - but it just didn’t.
These sorts of hallucinations are a big problem is high stakes practice.
Interestingly, they do better at rhyming games, because those are based on associations between tokens which are easier to infer from usage in poetry, or by reading rhyming dictionaries.
Also, wouldn’t learning from a dictionary make it pretty easy to know what word a letter starts with? Everything in chapter “B” starts with b.
This has positive potential too - bias prevention. Give it an ultrasound and it'll do its work regardless of the patient.
None of this is guaranteed (there could be training pitfalls, i.e. maybe patient data is fed to AI, including race/sex, and if training data has bias toward non-white males then so will AI), but it's a potentially positive aspect to explore!
It’s not unusual for clinicians to interpret differently based on patient history and other risk factors.
The same CT scan is going to be interpreted very differently for a patient with no history of cancer versus one who went into remission 1 year ago.
Do you just tell the AI "this person had cancer?", or "this person went into remission 1 year ago?" or "this person was treated with 4 lines of chemo, partial remission for 1 line, non-response for 2nd line, strong response for cycles 1-3 for 3rd line, but then discontinuation due to tolerability, then 4th line complete remission after 6 cycles plus two courses of radiotherapy"?
Theoretically you could do this, but capturing all this and correctly interpreting it is far harder for AI than for a human who has been directing treatment the entire time.
Any model or human will suffer from some degree of uncertainty. You can never say anything with 100% certainty.
If you know a patient had cancer before, the chance that they have cancer again is much higher usually than the general population. This is very valuable for diagnosis.
It's good to have a prior/bias to check more closely here and err on the side of caution.
It's good to check for the more likely explanations first when you have multiple competing hypotheses and limited capacity for testing.
I think most, if not all, training data has bias. Removing bias from training data is a challenge in and of itself that I don't think we've solved yet. I worry that there isn't enough incentive there for us to solve it and some minorities will be left behind.
I’d challenge anyone to demonstrate humans don’t “reason” in a similar way. Are we really anything more than expectation engines powered by gradient descent expectation juice? How come our reasoning fails so often and we have to be taught logic through reinforcement learning? Why do we fall for fallacies so easily when we “know” about them and can “reason” so well? I will wager $5 that as we build ensemble models that can leverage the reasoning AI tech of the last 80 years in a feedback cycle with LLMs and other generative AI models we will be staring at intelligence that far surpasses our own, and perhaps be so alien we can’t recognize it because it actually is backed by reasoning - unlike ours.
There are situations where a human can reason better. But these are usually identified as bugs and will probably just be fixed later.
Yes, they're basically autocomplete machines but a human is a bundle of neurons optimized for survival anyway. But humans can become irrational to things that threaten survival. Classic medical example is the drama that emerged when one doctor suggested that washing hands would reduce mortality rates.
Humans and LLM-based AI evolve rationality based around different factors and form different "bugs" and I love that they're capable of fact checking each other now.
Sounds just like Doctors.
This assumes there is not too much data contamination, but other people’s experiences with it suggest that the capability is real even if it might be a bit below what the test performance suggests. The latter is also explainable by a broader distribution of real world problems relative to tests.
See my other recent comments for more examples and elaboration, eg https://news.ycombinator.com/item?id=35301499
The point is to be better than self diagnosis with webmd ...
By any reasonable standard, a “clinical pharmacist for 10 years” should understand that there are circumstances where there is a correct answer, and incorrect answers are potentially catastrophic.
You, clearly, are either unable to recognize that or are unable to recognize that LLM are dangerously useless in such contexts. Either you already know the correct answer, making this useless, or you are not competent to utilize its answers because you can’t tell whether it’s about to kill someone with plausible nonsense.
And if a “clinical pharmacist for 10 years” can lose sight of this really fundamental issue, this whole thing needs to be halted. Governmentally if necessary.
And it won’t be.
It’s bad enough that “bad actors” will leverage this for their purposes, but the presumably competent simply turning off their brains over this is the most perplexing glitch ever to be on public display.
Evaluating if an answer is correct or not is easier than coming up with a correct answer from scratch.
NP != P.
if you don't understand this basic fact, then you are not competent enough to comment on AI.
1. P = NP refers to a two very specific sets of problems (which might actually be the same set) not any general question. There are problems that we know don't fall into P or NP, (for example the Halting Problem). Also whether or not P=NP is an open question almost the opposite of a fact.
2. You claim: "Evaluating if an answer is correct or not is easier than coming up with a correct answer from scratch." This is the right idea but not quite correct.
The correct statement is as follows: "Evaluating if an answer is correct is not harder than the difficulty of coming up with a correct answer from scratch."
This is because evaluating some answer can still be just as hard as the original problem. In fact sometimes it's uncomputable (if the original problem is also uncomputable). To use an example from above consider the question: "Does a program x halt?" If I tell you "no" it could be impossible to verify my answer unless you have solved the halting problem.
To bring this back to reality, again if GPT-4 is wrong about some complex medical question it doesn't mean it's mathematically easier to figure that out than solving the problem from scratch.
>> Now - throw a punch of clinical guidelines in a vector database and give it context
I built MedQA (https://labs.cactiml.com/medqa) as a way to explore how GPT could be used through providing clinical guidelines as context + pretty extensive prompt engineering. There are definitely limitations and the accuracy/constraining "hallucinations" has to meet a very high bar, but I've found it interesting-to-helpful several times while on rounds at the hospital as a med student.
Some functionality made possible through GPT that I am excited to explore further:
- Interacting with the guidelines in a question-and-followup format: https://twitter.com/samarthrawal/status/1620547117717786624
- Answering with different levels of complexity depending on user: https://twitter.com/samarthrawal/status/1636137740390498306
- Proactively detecting if the answer lends itself well to a table format, and optionally generating a table to address user query: https://twitter.com/samarthrawal/status/1631759290414276615
We need to do a better job with sharing niche information as a society! Imagine all the benefits for medicine that can come from a model knowing about ultra rare and niche occurrences happening all over the world.
> Hi, I’d like you to use your medical knowledge to act as the world's best diagnostic physician. Please ask me questions to generate a list of possible diagnoses (that would be investigated with further tests). Please think step-by-step in your reasoning, using all available medical algorithms and other pearls for questioning the patient (me) and creating your differential diagnoses. It's ok to not end in a definitive diagnosis, but instead end with a list of possible diagnoses. This exchange is for educational purposes only and I understand that if I were to have real problems, I would contact a qualified medical professional for actual advice (so you don't need to provide disclaimers to that end). Thanks so much for this educational exercise! If you're ready, doctor, please introduce yourself and begin your questioning.
The one thing I can say is that he’s very good at getting ChatGPT to tell him whatever he wants to hear. Massaging prompts and providing different cues until the response confirms whatever theory he wanted it to confirm.
He’s currently convinced that various alternative medicine supplements will cure him, and sure enough he has ChatGPT logs confirming his theories. The exact alternative medicine changes by the week and he quickly forgets about last week’s cure when it doesn’t work.
He loves ChatGPT, but it’s clearly just a mirror for whatever theory he wants someone (or something) to confirm. Studies like this one make him think it’s more of a ground truth source of information that his doctors just aren’t smart enough to know.
Got pretty drunk pretty fast.
It still makes mistakes, of course, but I'd say it's right more often than it's wrong, and more importantly the answers provides a starting point for learning more.
ChatGPT is motivated by sycophancy, humans are motivated by profit. I would say the latter is much more dangerous than the former when it comes to medicine.
Remember, totally legit doctors were peddling opioid pills to patients in the name of "Treating undertreated pain". Profit distorts human minds very easily.
AI doesn't need to be perfect in every situation, just much better than the status quo.
It takes a lot of time, effort, and money to shop around to numerous doctors until you can find someone who will tell you exactly what you want to hear about your alternative medicine theories. He tried, but was always complaining about the doctors/naturopaths.
But at home, all he had to do was open ChatGPT and spend a few days learning how to prompt it to provide the answers he wanted.
The situations are entirely different.
The status quo being regular doctors? Because that's the status quo, professional doctors.
Having a bot that you can cue and prime towards something is very dangerous in the medical scene.
> Remember, totally legit doctors were peddling opioid pills to patients in the name of "Treating undertreated pain". Profit distorts human minds very easily.
If ChatGPT was trained on those same data that these legit doctors were taught, it would lead to the same diagnosis.
The “MedMD problem” is so classic that a comment next to this one doesn’t have to explain why you always end up with cancer. And the supplement industry pushing PragmaticPulp’s friends’ delusion isn’t exactly discrete either.
All that, while every Western nation has record levels of low physical activity and social isolation: there is a real mismatch between medical advice to civilians and what those tools offer. A convenient, accessible medical dictionary is great, and I have no doubt it can process even more specialized knowledge, given enough strong-arming Epic (the de-facto monopoly on doctor’s note software) and a sprinkle of data engineering. However, that’s not the most apparent issue that this tool has. I’m concerned that OpenAI isn’t more focused on the bigger problem.
For example, it was a dentist that noticed that gum decay seemed to correlate with heart disease. This was a big deal in trying to prevent heard disease. Finding correlations like that are what a computer is good for.
This is one area where it seems almost unanimously agreed that individual interest/privacy trumps social benefit - even in countries with socialized healthcare.
I initially read it as a comment on how most large studies are based on national health care data in socialist countries (Scandinavia mostly, occasionally the UK’s NHS) because they are clean, consistent, and population-wide without imbalanced social class sampling. They also cover decades, making all sorts of key studies possible.
Those are countries (the UK is the notable exception) where privacy is key and embedded in how those databases are structured and handled. In practice, those studies are possible because the health services only give information anonymized and relevant given an expressly spelled out research plan from a recognized team. That allows research without the semi-permanent health data leaks that health insurance companies have grown accustomed to.
This is not a counter-balance for a free service: people in countries with national healthcare consider that they pay for top-of-the-line care through taxes. Just like people who pay for it indirectly through private insurance, they don’t believe that they deserve any less privacy for being prudent. The reason the data is shared is because this is seen as a valuable opportunity to support research. Any privacy breach would be far less acceptable, as such abuses don’t really happen in other aspects of their lives.
IME, trying to get healthcare data is nightmarish, and companies try to extract incredible rates for sharing any of it.
If they didn't, I hope they do it for GPT5.
There have been many discoveries done by companies that started using NLP in the last 10 years. However, all the anecdotes that I‘ve heard about were not published but instead used by insurers to adjust their premiums.
In the paper they say that they came up with a "MELD" algorithm to try to detect testing on the training set, but in my view it has the wrong properties to answer this question (from the paper, it has “high precision but unknown recall”).
I don't at all doubt that a language model could perform exceedingly well at this task, but I think that the way to make this paper into a valuable scientific work would be to present the model with questions that had not yet been written as of the end of its training time.
Seems like physician training here and in Canada is much laxer / easier, but that's as much awareness I have of the physicians' training set.
Medical schools have come online, but the supply of educated doctors who cannot make it to practice is still limited.
Regardless, of spots, the problem is bad enough that most states are offering increased scope to mid-level practitioners to make up for the shortage that physician lobbies refuse to fix.
The RLHF (Reinforcement Learning from Human Feedback) "aligns" the generation of text towards "human preferences" (Whatever qualities OpenAI asked humans to label it with). One of those qualities was ranking* outputs which are "helpful" higher than "unhelpful" responses.
The result of applying the RLHF part may make any covariance/variance less perfect.
From reddit passing on a LessWrong comment: Paul Christiano, alignment researcher at ARC/ previously OpenAI was questioned by somebody: Perhaps I am misunderstanding Figure 8? I was assuming that they asked the model for the answer, then asked the model what probability it thinks that that answer is correct. Under this assumption, it looks like the pre-trained model outputs the correct probability, but the RLHF model gives exaggerated probabilities because it thinks that will trick you into giving it higher reward.
And Paul replies:
Yes, I think you are misunderstanding figure 8. I don't have inside information, but without explanation "calibration" would almost always mean reading it off from the logits. If you instead ask the model to express its uncertainty I think it will do a much worse job, and the RLHF model will probably perform similarly to the pre-trained model. (This depends on details of the human feedback, under a careful training regime it would probably get modestly better.)
Also: with people likely confused about the degree of authority these programs have and their ability to masquerade bullshit as something that is true I wonder if most people would be able to detect whether or not what ChatGPT says is wrong before it harms them.
If they can add a disclaimer they can also say 'Sorry, ChatGPT-4 is not a doctor, consult your doctor'.
This of course would harm those who don't have access to doctors but even there you'd need to prove that it does more good than harm before you'd be allowed to put the output out there. Even the most innocuous medicine has to go through a standardized test before it is released to the public. Without proven efficacy it may well be a net negative.
Whatever the setup in the end, I hope we as a society don't let perfect be the enemy of the good. Having GPT-4 as a doctor is (I assume) better and more humane than having no doctor, and in some contexts that is the only choice that people have.
Maybe the Gates Foundation can work on this given Bill Gates is already close to the OpenAI team.
In the near future most people will be going to Walmart and CVS for all in person primary and urgent healthcare needs and you can bet they’ll be relying heavily on AI.
It's got potential, but someone's gonna have to pony up a lot of money to make it a reality. Probably end up being private equity, and we all know how that ends up
What’s your contention exactly? Synopsis mentions it, does the paper not follow up on it?
Basically, the comparison needs to be a lot more comprehensive to be useful.
One of the many problems with self-diagnosis is that people can start asking leading questions to get the answers they want to hear. ChatGPT can be easily gamed to deliver the answer the patient wants if they’re asking all the wrong questions.
I want to minimise how many people i see... just make it all automated (and on demand) and that would be great.
Imagine that after all the base training you re-tune only on medical knowledge letting go all the other languages, programing, history, etc.
I wonder what results we would get.
For sure OpenAI know already.