Comment by papeda | Hacker News Reader

Comment by papeda | Hacker News Reader

Here is a basic question: how are the sensitivity and specificity (false negative and false positive rates, IIRC) of these tests determined?

It seems like a learning problem with noisy labels. Does anyone know what they do in practice? Especially chaotic situations like this where there aren't reference tests? Are there reference positive and negative samples?

One concern also referenced for the Stanford study was that, as mentioned in a USC popular writeup on this study linked elsewhere on this post,

> Premier Biotech, the manufacturer of the test that USC and L.A. County are using, tested blood from COVID-19-positive patients with a 90 to 95% accuracy rate.

They must be factoring this into the 2-5% infection rate number somewhere?

In the Stanford study, they used blood samples that were taken before the epidemic to determine the true/false negative rate, and blood from patients that tested PCR positive in the past to determine true/false positive rate.

...which has been characterized as being analyzed in a highly suspect way [1], where only point estimates of the specificity were used.

[1] https://news.ycombinator.com/item?id=22924118

The Stanford study didn't properly include the specificity in their predictions. See here, https://statmodeling.stat.columbia.edu/2020/04/19/fatal-flaw...

Reference negative samples are relatively easy, at least in theory. Any blood samples drawn before the Covid-19 outbreak started are guaranteed to be 100% negative.

The sensitivity and specificity of these new tests is not established. It's part of why the FDA hasn't approved the tests yet. Scientists are doing their best with tests with unknown reliability, but it's a statistical mess right now.

Other more squishy factors are important, too: I don’t think Facebook posts for recruitment are a good idea. This could lead to more people taking part who have been sick or who had contact with someone who had the virus, over-representing that group of people in the sample.

Reply on news.ycombinator.com