Hacker News Reader

Comment by dahart | Hacker News Reader

I don’t know a thing about Holmes, I’m just reacting to your framing - don’t all medical tests that real people rely on have error rates? Doesn’t an 11% error rate suggest something that was real and mostly works but maybe wasn’t ready for prime time? The error rate could have been higher, perhaps. You only listed false positives, while the much more concerning problem would be false negatives - people who didn’t get scared or act when they should have. Still, a medical test error rate on its own doesn’t suggest deceit or fraud. People receiving terrifying but wrong results is a common problem that already exists in medicine.

Google tells me the FIT test for cancer has a false negative rate of between 8% and 26%. That appears to be likely worse than the Theranos numbers you cited here.

View on HN

All tests do have error rates and all doctors know this very closely because they look at results every day.

However saying “all tests have error rates, therefore all tests with error rates are equally bad” is a completely wrong takeaway. One of the things Theranos was doing was using traditional equipment to run the tests as a smokescreen for the fact that they didn’t have any working technology. So the results should have been similar to anyone else running them when that was the case.

But there is a problem that when a company is saying they’re doing one thing or using one technology but your actual tests could be run on unknown combinations of unknown lab hardware depending on the day and the whims of a CEO trying to defraud investors, you don’t know what kind of results you’re going to get.

Real labs have established controls and ship reference ranges for their test results because they control their processes. If Theranos was doing who knows what to keep the test results flowing to keep the illusion up for the primary purpose of satisfying investors, that’s not what you want to see out of your test lab.

> Google tells me the FIT test for cancer has a false negative rate of between 8% and 26%. That appears to be likely worse than the Theranos numbers you cited here.

You’re looking at the wrong thing. The Theranos number above is for tests they determined had to be voided because they weren’t run properly due to procedural errors they identified. Their own incompetence or fraud. And those are just the ones they admitted to!

The false negative rate for the test you Googled is a false negative rate, which is a simple fact that tests can’t reliably diagnose everything even when run properly. Tests look for markers in samples as a proxy for a condition, and those markers aren’t always present at levels that meet the thresholds even if the condition exists. Even when run absolutely perfectly, a test like that will have false negative rates.

The part you’re missing is that the Theranos voided and invalidated tests were due to problems on top of the normal test variation rates because they were from Theranos doing things wrong, which is directly downstream of Holmes manipulating investors.

> Theranos was doing was using traditional equipment to run the tests as a smokescreen for the fact that they didn’t have any working technology. So the results should have been similar to anyone else running them when that was the case.

> But there is a problem that when a company is saying they’re doing one thing or using one technology but your actual tests could be run on unknown combinations of unknown lab hardware depending on the day and the whims of a CEO trying to defraud investors, you don’t know what kind of results you’re going to get.

Exactly. They were using Siemens machines, but they often had insufficient blood because they were trying to maintain a pretense of "nano collection". Siemens' machines weren't even trying to pretend they could work like this, so Theranos kept trying to figure out ways to dilute or diffuse the samples they did have to get to a usable volume for the tests. But of course every experimentation there was destructive, so they had even less to use.

So we're not talking about error rates on properly calibrated machines, with sufficient blood that re-tests could be performed before needing another collection, we're talking about error rates on effectively misusing machines.

> therefore all tests with error rates are equally bad

I did not say that, nor even imply it.

Oh I don’t doubt there was funny business or fraud. What you’ve added is what the parent comment was missing; the reasons beyond the error rates.

> the test you Google is a false negative rate, which a simple fact that test can’t reliably diagnose everything even when run properly.

Yes I mentioned above it’s a false negative rate, right? And false negatives are potentially more dangerous for survival than false positives. Again, I was just reacting to the narrative about people getting scared by false positives, as if that was the primary problem. Your comment supports my claim that the error rate was not the primary problem, no?

Only 11% of errors were reported, there were probably many more.

Also, the standard for tests in the medical world is usually much higher than 11%, except for things that are life threatening for which no better test exists.

The error rate could be higher, which I mentioned already, but who knows how much higher. I don’t see how speculating about the error rate helps anything. Did you mean to say the error rate is usually much lower than 11%?

> except for things that are life threatening for which no better test exists.

This was precisely why I chose the FIT test example. There is, in fact, a more reliable test than the FIT test. Several, it turns out. Reliability isn’t the reason for the FIT test’s popularity.

I was unfamiliar with the fit test but after checking on it I guess its false positive rate is acceptable because it is used as a screening prior to a more invasive colonoscopy. In other words proper application of the test as a screening tool should result in fewer colonoscopies, not more.

The false positive rate isn’t the problem, the false negative rate is, which is why I mentioned the false negative rate above, and not the false positive rate. Obviously it would result in fewer colonoscopies and that’s the problem. The whole reason people take a FIT test is to avoid getting a colonoscopy. False negatives means a larger percentage of people will have cancer and not be aware of it, and not schedule a more reliable test until the cancer develops further and gets caught later by some other test maybe. The FIT test is better than nothing, but worse than getting a colonoscopy in terms of safety and reliability. It’s popular because it’s convenient, not because it’s highly reliable.

Reply on news.ycombinator.com