Comment by drcode | Hacker News Reader

Comment by drcode | Hacker News Reader

We need to start using all the big data muscle on basic research of cell chemistry, less on analysis of patient data.

I could probably fit all well-structured data elements in existence for all medical patients in the US for a year on a single DVD. (and even the unstructured non-image data is pretty sparse.)

This is not really a problem where "big data" techniques are the right tool- There's too little data. However, I believe we should be experimenting on cell cultures in laboratories at large scales. None of this is happening, I believe... no laboratory is running hundreds of thousands of cultures in parallel with carefully manipulated chemical environments and generating data from them.

A data set generated from such cell culture analysis could be petabytes in size. With that type of data set I believe it would be possible, using big data techniques, to get a lot of rigorous and causative details of chemical cell pathways relevant to disease formation that we currently lack. This is the direction I think we need to be going.

I certainly would love to be wrong and would love to get news that IBM has cracked heart disease with this new project. However, my guess is that this is a very inefficient way to apply big data techniques to cure diseases.

"I could probably fit all well-structured data elements in existence for all medical patients in the US for a year on a single DVD."

As someone who builds and supports a massive "big data" healthcare platform, I have to say you are horribly mistaken. Non-image well structured patient data, even if compressed, across all healthcare, with potentially multiple EMRs and data collection systems per healthcare organization, even if just the last years' worth of data, is "big data" and can benefit from "big data" techniques. Like all "big data" endeavors across any number of problem domains, health care application of "big data" is in its infancy.

Will the use of "big data" techniques over patient data discover disease cures? I don't know. But there is gold in them there hills... what kind? We will have to wait to find out.

I agree I might be wrong, but let me try to do the math (so you can improve it with your knowledge.)

Let's say 10 million hospitalized patients a year, 100 data points entered by a nurse, another 300 from lab tests during the visit, 50 entered by a doctor, 50 entered by a pharmacist.

Let's postulate 8 bytes per data element (most are numeric or ordinal.) This leads to:

10,000,000x8x500=4E10 bytes=40GB

Compressed, this would fit on a DVD (8.5 GB)

(I agree the numbers are somewhat larger if unstructured text is added to the mix, but it still seems to fall short of the terabytes of data I would expect for a "big data" approach.)

Usually I thought the "big data" moniker is applied to web sites where every minuscule site interaction is audited from every user across 10mil+ users, leading to much larger data sets. Or, it is used in the context of crawling the entire web where many many petabytes of data become available.

You might have better knowledge of the numbers involved and can correct me, please do.

One of the main issues is that cellular analysis in the basement is nearly impossible. I could actually see this coming to life if hardware was in the hands of the correct person.

Reply on news.ycombinator.com