Comment by Thriptic | Hacker News Reader

Comment by Thriptic | Hacker News Reader

I have very little hope for this succeeding, although I hope it does.

I say this because this problem is vastly more complicated than people believe for the following reasons:

* A substantial amount of data about patients is encapsulated in clinical notes which are free text. There is no standard format for notes; notes wildly vary in terms of quality, detail, and accuracy; and notes are full of non-standard abbreviations and clinical shorthand.

* A large amount of the data in an EHR is incomplete or incorrect. Information may be out of date or recorded incorrectly. Patient medication lists are a great example of this, and they frequently cannot be trusted by physicians even during the course of a single hospital stay (medications are started or discontinued without being properly recorded). Then there is the issue of non-adherence: patients frequently claim to be adherent to treatment regimens that they are not actually adherent to which is going to distort outcome data. In fact, the majority of people are non-adherent for many different treatment regimens. More generally, diagnoses may have been incorrectly determined and data may be incorrectly labeled or missing.

* Data exists all over the place even within an EHR, let alone across facilities. I myself have been to at least 5-10 hospitals and more outpatient clinics over my life, none of whom share data with each other. No one entity has all the relevant data about a patient, including insurers, specific care teams, or the hospital systems they got this data from.

* Data placed into a EHR may be intentionally falsified for simplicity or to get insurance coverage. It is not uncommon for physicians to intentionally misdiagnose a patient so their insurance will cover a diagnostic / treatment / procedure, or to input a less specific diagnostic code to save time when a more specific one is available.

Trying to train with data sets this fucked up is an enormous challenge.

I feel like Google have the power behind them to try to at the very least data mine the free text data for information. Possibly they even have enough sway to convince doctors and other individuals towards a form that is far less random, or at least to provide a platform to share information between patients. Note that NLP seems to have come a long way in five years, who knows where we'll be in a decade or more.

In terms of incorrect records, I'm not sure this would be a major problem over a large dataset. Making this up, but I imagine the range of errors is fairly random across all fields and with enough data you can always identify an approach that appears to save a life more than some other approach, even if the diagnosis is incorrect.

Although I totally agree with you that this is going to be an enormous challenge for Google, I feel like it would be underestimating them to assume this has very little hope of succeeding.

I agree that if Google throws enough money and talent at the problem then they can certainly make improvements, but I seriously doubt that they are going to create some sort of revolutionary clinical decision support system which is going to massively improve outcomes.

Bingo. Healthcare data is so unorganised its not even funny. That's often why you see machine learning medical leaps in really niche areas, like making a diagnosis based off an MRI scan etc. I think if this does succeed, it will succeed in an equally niche area, such as with a specific system, in a specific clinical environment.

All of these points are not fundamental problems. Researchers are not writing expert driven rules where the "raw" accuracy of structured data is that important.

The only important part is making sure that the training labels are accurate. If the training data has inaccuracies, the model should learn to work around them (with some accuracy cost). The main question simply is how much signal is in the data. And that's hard to predict in advance (see the recent surprising results about predicting sexual orientation from faces).

>> And that's hard to predict in advance (see the recent surprising results about predicting sexual orientation from faces).

Those results were not so much surprising, as coerced. For example, training data included an equal amount of men of both sexual orientations and the same for women- an equal distribution, in stark contrast to the one in the real world. This was clearly done to overcome the inevitably poor accuracy when training with unbalanced classes.

In short: don't base anything on that paper, for it was a steaming pile.

Reply on news.ycombinator.com