Hacker News Reader

Comment by aix1 | Hacker News Reader

One thing that jumps out is that the paper fails to provide a proper direct comparison with Med-PaLM, only mentioning the latter in passing almost as an afterthought. Looking at arXiv timestamps, the Med-PaLM preprint has been out of three months (given the rate at which this field moves, an eternity). I understand Med-PaLM the current state of the art and would be a natural point of comparison.

https://arxiv.org/abs/2212.13138

View on HN

> Our results show that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models (GPT-3.5) as well as models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM 540B).

What’s your contention exactly? Synopsis mentions it, does the paper not follow up on it?

Med-PaLM has been tested on a number of public benchmarks, and the results published. A useful comparison would have been to test GPT on the same set of benchmarks. Yet the paper we're discussing offers just one datapoint by way of a numerical comparison, literally in a footnote.

Basically, the comparison needs to be a lot more comprehensive to be useful.

Late to the thread here, but the paper announcing Med-PaLM (https://arxiv.org/abs/2212.13138) does not report many benchmark results on Med-PaLM and is instead mostly about Flan-PaLM 540B (which is compared against in this paper). I am curious if any other Med-PaLM benchmarks have been published but I don't believe it is currently possible to do any further comparison against Med-PaLM given that the model is not public and no other open benchmark results are reported in the original Med-PaLM paper.

Reply on news.ycombinator.com