Hacker News Reader

Comment by macksd | Hacker News Reader

I was a bit confused by exactly what the percentage in the headline was, so here:

>> In two of the largest retrospective cohort studies of AI to replace radiologists in Europe (n=76 813 women), all AI systems were less accurate than consensus of two radiologists, and 34 of 36 AI systems were less accurate than a single reader

I'm still a little unclear how you measure the accuracy of a consensus of radiologists, without relying on the consensus of other radiologists. Maybe just a bigger consensus, or by using known end results but looking at imagery that could have been an early warning?

View on HN

The methodology in this paper compared the results of the radiology report to a follow up biopsy.

Or inversely, can this be interpreted as 2 of 36 AI systems are better than humans?

Therefore Hospitals should be investing in those 2 systems?

Only if you started with a hypothesis that those two AIs were the best. If you did not, then you're just p hacking.

Or that 2 of those systems got lucky, none of them actually work for their stated purpose, and investing in radiology AI is about as wise as investing in the Keely Motor.

Unless the next time you test, it's a different 2.

Not necessarily. If you were running a casino and someone was able to significantly beat the odds on 34 of your 36 tables, would you assume they cheated at all other tables but not those two, or just that other players at those tables were really good?

It could be that 2 of those systems are actually better. It could also be that the state of the art in deep learning just isn't there yet for radiology. But this is a strong suggestion that research is being dramatically overstated, so I would get real careful with my investments in this area overall.

We've since changed the title. The submitted title ("94% of 36 AI systems evaluated were less accurate than a single radiologist") broke the site guidelines by editorializing. Submitters: please don't do that.

"Please use the original title, unless it is misleading or linkbait; don't editorialize."

https://news.ycombinator.com/newsguidelines.html

Well, to train the AI you would be using radiologist reads anyways so I'm not sure that's really a problems.

Oh yes, just that easy! Here, AI, he's some radiologist reads. Learn them, we're going live in Q1 2022.

You could simply see if the person had actually gotten cancer with the benefit of perspective.

Reply on news.ycombinator.com