Big data in genomics: The $1k genome has arrived | Hacker News Reader

Big data in genomics: The $1k genome has arrived

nature.com

85 pointscedricr54 comments

The bottleneck in genomics hasn't been cost since about 2012.

The chokepoint is analysis. Most biologists don't know how to program, and most programmers don't know biological context, neither know statistics well.

I work with so many scientists whose only thought is to sequence first and ask questions later. Usually all the real work ends up falling on the shoulders of one skilled researcher while the rest look on like some unionized road crew.

It's only going to get worse, but the good news is, if you are one of the biologists who can program and use statistics then you're in good shape. There is already so much idle data out there already, that you'll never have to spend a dime on sequencing.

You know... I'm not so sure I completely agree. I mean, yes, the analysis is the bottleneck. Since about 2012 it takes more time to perform the analysis than it does to generate the data. And even though all of the raw data -> BAM -> VCF or counts steps is automated, nothing after that is (or can be, since that's where the science happens).

But, the $1k genome really is significant. Costs matter.

Researchers will only sequence what they can afford. So, when sequencing a genome was $10k, there weren't too many 30X whole genomes sequenced. Instead we had exome sequencing of individuals or low-depth sequencing of populations.

Fast forward a few years, and now the costs have gone down significantly. Because of this, we are now starting to see more whole-genomes. Now, the costs of to sequence a whole-genome at 30X is about the same as targeted exome sequencing (due to the extra capture costs). If it's almost cheaper to just sequence the whole-genome and skip any extra capture probe biases, why not just sequence the whole thing? This is particularly true when looking at complex diseases where the underlying cause(s) may not be in the coding part of a gene.

So, in that regard, cost really was a bottleneck. With all of the new whole-genomes available, we can start to ask even tougher questions. And with more data available from more people, we can start to get some real confidence behind our predictions.

But, you are right in that it does all lead back to the data-analysis bottleneck. Before we had as more data than we could deal with. Now, we can process data faster, so that's good. But we also have significantly more data to process, so in the end, it's about even.

The other issue then becomes data storage. What do you do when you have a few hundred terabytes or a petabyte of sequencing data? S3 can get pretty expensive at that scale. Instead, it's probably just cheaper to store the DNA in a freezer, and if you ever need the data - just resequence it.

Edit: reworded a bit

>With all of the new whole-genomes available, we can start to ask even tougher questions. And with more data available from more people, we can start to get some real confidence behind our predictions.

What I'd like to add to this is that we can spend more time and less money on "hobby" projects!

These are the projects you don't really have explicit funding for, but you do anyway, using leftover money from various pots [1]. We have two PhD students who work on "hobby" projects.

These projects include "outlandish" questions or "basic" research the funding committees didn't see the reason to fund.

In our case, these two PhD students work with less studied plant organisms no-one else really looked into so far, but which have some interesting attributes. Some of these hobbies have been fruitful (interesting novel salinity resistance proteins found), some of these less (but it's always good to populate public databases from data of less well-known organisms).

Cheaper genomics allow us to do so much more of these projects, exciting times!

[1] See http://www.phdcomics.com/comics.php?f=1431

(Of course, now the cost bottlenecks move somewhere else - it has become cheap to sequence an entire plant population, but it's still expensive to grow & especially phenotype that population - if you're into computer vision and are looking for a new project, automatic plant phenotyping is a field that still has a lot of work to do!)

Compute time and storage costs are also bottlenecks, so even though the base calling through VCF is automated large genome centers struggle to keep up with the output of the new sequencers

Right - but we used to struggle with the output of the last generation of sequencers too. Computing capacity and storage have scaled at about the same rate. Or perhaps it's more accurate to say that the costs have come down, so we can throw more iron at the problem - these are embarrassingly parallel problems...

As an example, the last sequencing analysis cluster I setup (~2010) ran with 2x bonded gigabit networks between our compute nodes and storage and had 10TB of network storage. The newest one is running 10GigE, has 200+TB of storage, and is cheaper. And we are a small shop in this field.

When you say compute time, do you have in mind purely quantitative processing/analytics, or streaming and data transfer?

Is there any way you could describe a sample "experiment" and the associated storage costs and compute time, just a paper-and-pencil estimation perhaps?

I went to a python talk where the guy did genomics . I asked him how long his program would take and he said I'll tell you when it's done. This means his analysis programs can take days to weeks and he really didn't know how long the current one can take .

Possibly C. Titus Brown? Here's his PyCon 2014 talk, "Data intensive biology in the cloud": https://www.youtube.com/watch?v=RCRbfKK57X8

People say this all the time, but with some of the most common applications of high throughput sequencing there are very good canned solutions (using open source software) that you can pay for. DNAnexus, Seven Bridges, and Illumina BaseSpace all provide cloud storage and analysis. Unless you are doing a custom prep for your sequencing one of these probably has an analysis solution for you.

>DNAnexus, Seven Bridges, and Illumina BaseSpace all provide cloud storage and analysis.

"Analysis".

There is a difference between meaningful analysis which answers long standing questions of biology, and running a program which calls SNPs and spits out genome stats.

The hard part is knowing what questions to ask, and how to interpret the answers when you get them. That takes years of training to develop a critical mind.

This is absolutely true. I've done differential expression analysis on many RNA-seq and microarray datasets, and despite the fact that the basic pipeline for such analysis is quite straightforward and easily automated, every analysis is asking different biological questions and has different confounding factors that need to be addressed, so the analysis of each dataset ends up being quite different from all the others.

Analysis is custom if you are doing genomics right at this time. Either you are doing something pre-clinical, and you better understand what you are doing i.e. analysis in a black box is not ok. Or you are not dealing with human stuff at all and then there is no analysis in a box on the market.

There are tools with GUI's but they don't help that much if you don't understand the underlying systems.

I don't agree with you. I would argue that Genomics England is doing genomics right with the 100k genomes project and they certainly aren't doing custom analysis. Their pipeline is standard through variant calling and annotation. The WGS pipelines are pretty well established at this time. What you do with your VCFs is what can be customized. There is no reason for a small lab to write their own aligner or variant caller. The most that many do is write wrappers to create "customized pipelines", which is really just reinventing the wheel.

EDIT: I want to clarify my mindset. In the environment that I work in there are three stages of analysis: primary, secondary, and tertiary. We mostly ignore the tertiary stage (I'm not actually doing science). Primary analysis is on instrument (base calling), secondary analysis takes you through variant calls and annotation, and tertiary analysis is where the real science happens. When I'm talking about canned analyses I'm talking about what I refer to as secondary analysis. Tertiary analysis is a very hard problem and will often require custom solutions. In my mind I'm separating bioinformatic analysis (secondary analysis) with interpretation (tertiary analysis).

Getting the variants out is where the analysis starts. What does the variation mean, which ones correlate with a detectable phenotype. Which ones are affecting regulatory systems.

Genomics England is brilliant because it has good access to excellent NHS data which it will use to do a lot of analysis. Which its not getting off the shelf but a lot of talented clincal+bio+informaticians are working hard on that.

Or in other words getting the variants to the human genome reference is just the start of the analysis not the end.

i.e. I am more worried about the end of the pipeline than the contents of them.

I suspected as much on the 1,2,3 level. But even 2 is not really fixed yet. There are great off the shelf tools but even the concept of variation calling against a reference linear genome is not perfect yet especially with the alt_loci business. Which driving the variation graph work in the GA4GH projects.

It's not strictly true that we're not doing custom analysis. It must also be understood that while Genomics England have some standard pipelines running we are also providing the capability for partners via our GeCIP programme to work within a walled garden to do custom analysis within domains for rare diseases and cancers. You can read more about that here: http://www.genomicsengland.co.uk/about-gecip/

Is the time required to upload data to the cloud ever a problem with these solutions? Of course, it depends on what you are trying to do, but suppose you were working with thousands of genomes?

The sequencers can stream to a data analysis center as the data is being generated.

It takes a 100mbit stream/$1M of sequencing capital, so network connectivity to transfer to a data center is a tiny tiny cost of the whole ordeal.

However, paying for AWS storage is pretty prohibitive, unless you're at a small scale. So big centers will build their own storage facilities.

The small data producers like the ones that the thread author talks about can use often use AWS more cost efficiently than building a compute cluster. However, they need to budget for that, which is not always thought of. They may also need to fight their institute's core center so that they can use DNANexus.

S3 storage is pretty cheap, it's the data egress that really costs.

For academic centers though there is often an incentive to move things in house due to different treatment for capital expenditures and the opportunity to externalize some of your costs from your grant onto central services.

Data transfer is less than a single year of Glacier storage, so while it's pricy I wouldn't egress a major portion of the cost.

Keeping this data for less than 5-10 years is pretty questionable, since it's so expensive to generate. Eventually it may be cheaper to store the DNA and resequence when if it needs to be looked at again. However, if you're doing petabytes of storage, it's going to me much more economical to have your own storage and compute than to use AWS. Particularly at the rate that academic centers pay for sysadmins.

Running a public data portal our egress is higher than our storage costs. (We now proxy downloads through a direct connect to our university network...)

Remember to account for future reductions in storage costs. S3 has come down from $0.1500/GB month in 2010 to $0.0300/GB month today. And the recently introduced infrequent access storage tier is under half that again at $0.0125/GB month. It's now significantly cheaper to use S3/Azure/Google than running the storage ourselves.

Sure, all the time. Network and I/O are the biggest blockers for sequence analysis. For any organization that is working with thousands of genomes they probably have their own compute resources. I know of at least one organization who is currently sending thousands of genomes to the cloud for analysis, so it's certainly feasible to some extent.

Any chance we could follow up on this? I'm conducting research in this space and would love to have a chat regarding barriers to such large-scale analysis.

If you could message me at hngenometemp@forward.cat that would be terrific!

All of these will give you some answers. The reality is there is SO MUCH MORE INFORMATION to be gleaned from our genomes that these solutions do not account for.

Having worked in a genetics lab as a sysadmin, I can agree with this 100%.

Though that $1000 produces 100GB of data, after processing there's probably only about ~100MB of features left for machine learning, at most. And most of that will be incidental. With enough data we can hope to have a better filter between signal and noise.

Until now, biology has had a huge problem that most big data settings don't: far far more features than labels. With enough patients' data, the matrix will become squarish, but that's a long time from now, still.

>Though that $1000 produces 100GB of data

Not sure if you are referring to the raw sequence data coming from the sequencers, nowadays the standard practice is to align a specific sample's genome (e.g., an individual's genome) to the reference genome (the Human Genome Project) and store only your variations against that reference genome, stored as a BAM (binary alignment) file.

Furthermore, for most clinical cases, if you only care about the current known SNP (single nucleotide polymorphism in coding genes), you can generate a VCF file at those known SNP sites, compressing the data further. This is the way most population genetics is done by comparing the VCF files of cohorts, not doing analysis of single genomes of patients one by one.

You've got a bit of misinformation in here. He's correct about the raw sequence data for a 30X whole human genome being 100GB of data. The standard practice is to align the sample to the reference, but we store alignments in the BAM file, not variations. Some of the mismatches in the alignments are due to noise in the sequencer. We run the BAM through a variant caller which outputs a VCF which contains the variants. You do not typically genotype at only known SNP sites when generating a VCF file (although you may restrict by a region). If you were only interested in certain SNP sites you would be better off running a microarray.

Of course its a VCF file which we hope contains the variants ;( the tech is still fickle and so is some of the biology...

Soon we are going to go towards variant graphs and VCFs will disappear again, slowly (IMHO).

I found this link: https://medium.com/precision-medicine/how-big-is-the-human-g... In summary: > 1. In a perfect world (just your 3 billion letters): ~700 megabytes > 2. In the real world, right off the genome sequencer: ~200 gigabytes > 3. As a variant file, with just the list of mutations: ~125 megabytes

If there is only .1% variation then we should be able to get a diff down to ~1MB with some cleverness.

That sounds quite feasible, though it hasn't really been worth the effort until we have quite a few more genomes. And typically extra information about the variant (is it in a gene, does it change a protein, etc.) so that extra lookups aren't required during a scan.

There are typically 4-6 million variants discovered through this method of genome sequencing in a normal genome. A simple variant consists of a genome coordinate at ~32 bits (one of 3.2e9), and the change from the reference, which is a x,y index into {A, C, G, T}^2, at ~4 bits. Typically the coordinates are spaced on average ~1k bases apart, so the coordinate could probably be squeezed into ~15bits with clever encoding. So a naive encoding of this information gets to 27MB, and that could probably be shoved down into 10MB if coordinates are deltas from the previous, rather than absolute. 1MB seems feasible, but with diminishing returns computationally.

Interesting - wouldn't you need to have access to "the rest" of the genomes that you are comparing against? In other words, wouldn't you need to keep that ~100 GB from the newly sequenced genome in temporary storage while comparing against the rest of the database stored somewhere in the cloud, before then condensing the new genome into a variant file?

Typically we don't look at other genomes while we find the variants in an individual genome. Each genome is analyzed against the "reference" human genome, which is an average of 10 individuals. This forms the coordinate basis that is shared for everyone else.

Pretty much all genomic data uses a reference genome as the basis. This is versioned, and has a bug tracker, etc., for various regions that have been difficult to assemble.

The flow is:

1. BCL (scans of the glass slide) 2. FASTQ (individual short reads and quality scores, unsorted and in random order) 3. BAM (individual short reads aligned to the reference genome) 4. VCF (the "diff" vs. the reference genome)

All of this can be done with <10GB of reference data and code, where the reference data is the current human genome, a burrows-wheeler transform of the human genome, gene locations, and dbSNP (the database of common human variation).

How long does it take (and with what computational bandwidth) to produce a 125MB variant file from 200GB raw sequence data?

Depending on the pipeline you use and the compute resources available you could have a full workflow done in anywhere from several hours to a couple days. Illumina BaseSpace is free (for now) and has some example data sets with a bunch of canned pipelines for analysis if you're interested in trying it for yourself. https://basespace.illumina.com/

You're not going to VCF on a whole genome in several hours.

With particular hardware and software you can. Edico Dragen claims speeds for bcl -> vcf of 20 minutes [1]. With Microsoft Research's snap aligner and 450GB of memory you can get whole genome alignment in ~30 minutes and then variant calling can be done in a couple hours.

1. http://www.edicogenome.com/dragen/dragen-gp/

Could you please elaborate on this?

I've seen 200GB runs take 4 days, I've seen runs take 3 hours. Depends on your computing structure but more importantly is your IO. High CPU core counts and high speed storage access make a big difference, as does distributing the computational workload.

It's hard to quantify how much data is left after "analysis". An assembled genome sequence, at least a certain quality, is actually useful beyond machine learning.

For example loss-of-function mutations can be detected without resorting to other genomes beyond the reference. Inversions, translocations and copy number variations can also give clues to illness.

If for 50% of the genes (human) we have no idea of function, then its hard to determine at this point what is a loss-of-function mutation in the first place.

Source: I am the lead web developer for uniprot.org, and I deal with lack of real info daily as does everyone in the Swiss-Prot team.

Actually, when a patient with a mysterious illness lacks a functional version for a particular protein which healthy members of his family do have, this is a valuable clue. It narrows down the search significantly, and there have been case reports on this.

Of course, not every mysterious illness can be solved this way...

Apart from data privacy and formal restrictions on sharing, what would you say the reason for this lack of public data is?

Am I right to say that the bandwidth and storage needed to upload whole genomes is prohibitive?

We collectively just don't know. Not even in the cutting edge literature. Humanity, only has a basic understanding of what the biochemical role of many proteins is, and due too that also a similar idea of what the genes role is in the rather complicated system of making a functioning human.

Bandwidth and storage are infrastructure issues that could be solved with enough money. In biology we are still lacking good technical solutions to do the chemistry and even something "simple" like getting a protein crystalized so we can determine its 3D structure is not trivial or cheap yet.

Fast genome and RNA sequencing are massive improvements and really help. But the basic understanding of what all those genes, regulatory and other parts do is still relatively primitive.

We often "complain" that we wish that people stopped sequencing and went back to do more biochemistry for functional characterization instead ;)

Actually, can I contact you @uniprot.org with more questions?

http://www.isb-sib.ch/fp/people_search.php?cn=jerven%20tjall...

What I find most worrying with mass-collecting fully sequenced DNA (or DNA at all) is that it is inevitable that law enforcement, military, spy agencies or politicians will want access to the data.

Given enough sequenced DNA and a free-fall of sequencing costs, it might be feasible in the future to find out who exactly took a dump in public just due to DNA. And well, the Israelis are already doing this with dog dumps (http://uk.reuters.com/article/2008/09/16/uk-israel-dogs-idUK...), so applying the same tech to humans is not far away.

Fucking scary if you ask me.

I think the currenntly more interesting application in sequencing is pathogen detection. These have smaller and simpler genomes (mostly), and tracking them and their features improves epidemiology and the choice of treatments.

Pathogen identification is indeed a very exciting application for NGS. In case you're interested, here[0] is a paper about a tool called SURPI (Sequence-based Ultra-Rapid Pathogen Identification) which was designed for that purpose. Also, here[1] is a case report from the NEJM where SURPI was used to diagnose a patient with Neuroleptospirosis, which allowed him to be treated quickly and eventually recover. SURPI isn't the only horse in this game, of course, but I've worked with it before so it immediately came to mind.

[0]: "A cloud-compatible bioinformatics pipeline for ultrarapid pathogen identification from next-generation sequencing of clinical samples" http://genome.cshlp.org/content/24/7/1180.long

[1]: "Actionable Diagnosis of Neuroleptospirosis by Next-Generation Sequencing" http://www.nejm.org/doi/full/10.1056/NEJMoa1401268

The clinical work they're doing is great, but the code is problematic. Its a bunch of Perl and Python duct taped together with shell scripts. From the github repo:

Shell 84.3% Perl 8.9% Python 6.3% C 0.5%

Check out the source https://github.com/chiulab/surpi

nextflu.org and wwarn.org and ebola.nextfu.org

NextFlu takes global samples uploaded to a global flu genomic sample database and construct a phylogeny tree to monitor how influenza evolve and I think the authors behind the project wants to make predictions as to which cohort of influenza variation will become dominant.

Wwarn.org I believes tracks the emergence of Artesminin resistance (front-line drug in malaria treatment) in malaria in SE Asia and tries to map it out on GIS to inform public health officials further from SE Asia how it is spreading to their region (India, Africa where current Artesminin resistance gene is only 5% while Artesminin resistance is already the dominant wild type in SE Asia) and whether to modify front-line treatment protocol.

A quick search for public genome data led me here http://www.completegenomics.com/public-data/69-Genomes/. I wonder if there will be a time when you can search millions through a rich UI, comparing your own genome with others, etc.

For more genomic data check out www.solvebio.com

You'll need to register for a (free) API key to access public data, although we're removing that requirement in the next few days (I work here.)

Have a look at ExAC, which has 60k exomes and a pretty snazzy UI: http://exac.broadinstitute.org

Reply on news.ycombinator.com