Hacker News Reader

Comment by dvt | Hacker News Reader

> There is enormous potential for this new wave of tech to impact humanity. For example, Google’s MedPaLM2 model outperforms human physicians to such a strong degree that having medical experts RLHF the model makes it worse (!).

Wow, what a ridiculously disingenuous cherry-picked claim. If you actually read the paper you'll find this gem: "However, for one of the axes, including inaccurate or irrelevant information, Med-PaLM 2 answers were not as favorable as physician answers." Typical AI hype blog post donning the HN front page. At this point, I'm ready to put on my tinfoil hat and say that a16z, etc. is heavily pushing all these narratives because the next round of investments for the great majority of AI startups will almost certainly be the bagholder round.

View on HN

The author picks an example of an aspect where Med-Palm2 is impressive, and then your counter argument is to find another cherry picked example where the performance is not satisfactory. Great discourse..

There is nothing disingenuous about the post. Is your argument literally that there is not enormous potential for impact with this technology? And you're going to go with the argument that there's a single axis on which an ML system underperforms a frigging doctor? In what universe is an ML system (slightly) underperforming a human doctor on a single axis not one of the most impressive things to have happened in the past 50 years?

> a single axis on which an ML system underperforms a frigging doctor

You do realize the salience of the fact that this isn't some arbitrary axis, right? It just so happens to be the axis that involves the doctor making the correct diagnostic. Which is, you know, the entire reason doctors are a thing.

> slightly

I wouldn't call a factor of two "slightly," but that's neither here nor there.

Are we reading the same paper? In the graph I'm looking at the axis where the model underperforms the doctor is labeled "No inaccurate/irrelevant information", which has nothing to do with making the correct diagnostic.

The three important axes "Answer supported by consensus", "Possible harm extent = No harm" and "Low likelihood of harm" it is performing really similarly to the doctors, probably similar to the graph a single middle of the pack doctor would have.

Are you reading a different graph or am I misunderstanding something about it?

I think the axis OP is looking at is “more inaccurate information” which medpalm does perform more poorly on.

20 years ago i saw a documentary on Discovery Channel about AI. One case was about an AI used to diagnose measles. It worked very good, even with an old yellow car with flecks.

Felt the same: the Internet is now full of "I saw it coming, and can predict the future" type of blog article, where you could replace AI by block chain (or any new technology - superconductors?) and write the same article. Very bland and not worth of HN time...

Maybe. But then look at how fast accuracy is improving. I would bet the next iteration will be better in all relevant metrics.

Wow, that would be quite the step change in one iteration. Bold statement.

Not really. MedPalm 2 is already better or equivalent at all metrics except the one OP cited. So it will be just a reasonable improvement.

Reply on news.ycombinator.com