I took a look and zipped FASTA files are about 800MB. Large, sure, but people are streaming Netflix movies far bigger than this every night.
It's really hard for me to see the MPEG-G effort as anything but a money grab for compression patents and licensing.
500,000 Humans is a small sample of the UK population one company is looking to create.
All this is in the article.
500k humans would make this 400TB of data. Really not that much, especially compared to the $500 million dollars it would take to sequence all those humans at $1,000 each.
FTFA: "Typically, a DNA sequencing machine that’s processing the entire genome of a human will generate tens to hundreds of gigabytes of data."
Plural tens = 2to3 x 10 = 20-30. Plural hundreds = 2to3 x 100 = 200-300. Median[20,30,200,300] ~= 100 GB.
Why does whom need the original pre-calling measurements?
I get that the reads will be randomly distributed. Let's pretend that for a given position in the genome, the number of reads covering it are poisson distributed: i.e. the number of reads is chosen such that with sufficient probability each region is probably covered by one or more reads. This means the peak of the poisson distribution is much higher. So most regions will have over-redundant amount of coverage compared to the aimed for minimum global coverage. So the bulk of the data are actual repetitions of a single underlying sequence in the genome. Why can't these regions covered much more than minimum coverage regions, be called with sufficient certainty to discard the raw measurements? then only the low coverage regions may need raw data.
For most research purpouses it seems the raw measurements would be unnecessary? To the extent they are necessary we are admitting that we can't affordably sequence genomes yet (if you want I can sequence yours with my dice).
https://bitesizebio.com/8378/how-much-information-is-stored-...
Even at 1.5 GB, this doesn't seem like an insurmountable problem at the current cost of storage.
750,000 GB = 750 TB, and you can buy an 8 TB hard drive for $150. So for $15,000 you can store 500,000 human genomes.
If $15,000 is too expensive for a company, that company is doing something very wrong, when any salary will be many multiples of that.
Also the more you compress, the longer the processing time. Some jobs took days, so there was a trade-off. I'm not a genome expert btw, just a devops nerd on a related project.
As others have mentioned [1], the raw sequencing data (still) has to be stored, so 1.5GB is off by about 30x. Even at 6:1 compression, that's still 5x.
More importantly, though, the cost of storage, especially at scale, is significantly higher than just the price of bare drives. Backblaze famously complained about this [2] in 2009.
Even their particularly cheap, particularly low-performance solution claimed to add almost 45% on top of the cost of the bare drives. A higher-performancd storage system (using, say, standard SAS expander backplanes instead of their non-standard SATA multipliers) has, in my experience, added 70% on top of the bare drives (and that's being frugal, which seems remarkably rare, even among startups).
Your $15k is now $128k, minimum. That's just purchase cost, and, while operating cost isn't huge for that little storage, it's not zero. The cost gets much, much worse, if the orgnization can't/won't hire someone like me or won't allow that someone to implement such a frugal DIY assembly (which is the vast majority of organizations today).
That leaves "enterprise" storage or cloud, which are both approximately as expensive. I recently heard an estimate of half a million dollars per petabyte for NetApp, which would translate to $1,875,000 here.
[1] Well detailed by https://news.ycombinator.com/item?id=17821523
[2] famous on HN, anyway. https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-...
Wow.
But, yes, even for my lower storage numbers, for "EU (London)", even for just 3 years (a conservative enough lifespan for an HDD), S3-Standard would be almost $2.97M, S3-Standard-IA would be $1.77M, S3-OneZone-IA $1.4M, and Glacier would be merely $608k. Of course, that excludes transfer charges.
You're very correct that salary is often the single largest expense - true even in academia. This is consistently reflected in grants and other funding awards. However, a hardware expense of $15,000 would indeed be out of reach of many biology labs. How can these both be true?
Simply: graduate students are responsible for the vast majority of the lab's research output.
Graduate student salaries in biology are very far from many multiples of $15,000. In fact, the median salary is certainly less than the lowest multiple of that. The NRSA, (a coveted, extremely competetive grant well-beyond the reach of the majority of graduate students) pays $24,324 per year. [0]
For another example, the University of Florida publishes their stipends [1] - but if you are unfamiliar with the (always) distorted presentation of the numbers, you'll have to take my word on the specifics. The standard appointment will be at .50 FTE for 11 months, a combination you won't find listed. Such a researcher's compensation for the preceding 12 months would total $20,078x11/12 = $18,404, and that's before almost certainly some $500 of mandatory fees and other unsubsidized non-tuition expenses that all graduate students get to pay tax on but immediately hand back to their employer (university).
It's not much better in academia even after becoming a doctorate. Salaries in biology for post-doctoral (ie, already possessing PhD) researchers are about $36,000. [2] And that's after a delay averaging 7 years [3] earning 1/2 to 2/3 that figure at best.
Those numbers are perfectly in line with graduate student compensation here at the University of Illinois, for what that's additionally worth.
[0] https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-1...
[1] https://hr.ufl.edu/manager-resources/recruitment-staffing/hi...
[2] http://www.sciencemag.org/careers/2015/12/new-phd-incomes-su...
[3] http://www.slate.com/articles/health_and_science/science/201...