636 karma · joined July 22, 2016
[ my public key: https://keybase.io/danielecook; my proof: https://keybase.io/danielecook/sigs/fmxN1m8INHaWQOZUHDloBzwtH7-3yeAD33ZCTFnjtnQ ]
2688d4dbdddca1e4423286cffb82193b81dfc5f7dfdbdc81c2bd73fc0a7b056e
It also tends to be very portable.
And in the realm of programming, it’s often easy to take shortcuts even though later on they wind up costing more time for you or others involved: omitting documentation, not making something reproducible, poor code quality, etc.
Having spent most of my career in academia I see a lot of the work makes this trade off: efficiency is sacrificed for a sloppy, immediate “speedy” solution. You need to find a balance and think through problems, considering not only what is immediately in front of you but also what you’ll be doing in a month or year. Otherwise you’ll wind up with a project or codebase that begins to drag your productivity and output with it.
Remote: Yes
Willing to Relocate: No
Technologies: Bioinformatics, NGS, Bash, Python, Nim, Docker, SQL, Redis, Flask/Web
Resume: https://github.com/danielecook/cv/raw/master/cv.pdf
Email: danielecook at gmail dot com
A good workflow manager builds on these ideas further by managing environments, job submission, parallelization, cloud/cluster submission, and other options that make processing large amounts of data a lot easier and more efficient.
This looks useful, but can it submit jobs to cloud compute clusters or HPC systems and operate locally? Maybe I’m missing the point in terms of the purpose.
- combine commuting with working out. Even if you only run 1/2 way and still have to catch a train for the second half you’ll still be getting time back. - hire a cleaning service if you can afford it. Maybe a Roomba? - move closer to work to reduce your commute time. - don’t keep your phone by your bed. If you are like me you’ll waste time in the morning and evening. - don’t make more chores for yourself than necessary. For example, you can often wear pants 2 days of the week without issue.
I recently switched jobs and am moving next month to reduce my commute. I'll have an extra 7 hours a week to do whatever I want and I cannot wait.
If you only need a single SNP or a group of SNPs within a region, you can use tabix[1] to index gzipped TSVs and query by genomic position. The position of SNPs can be obtained from a lookup table (2.5M is not very big even for R) or from an API if you were to say - query by rsid.
tabix also works over http (and s3) and can utilize RANGE queries to select a subset of a file...so you only wind up downloading a or reading a small portion once it is indexed and can do something like this:
tabix https://www.file.url.tsv chr1:1-1000
The command above would return variants on chromosome 1 between 1 and 1000.
The following variant browser works in this way: https://elegansvariation.org/data/browser/ - Theres no formal database (e.g. MySQL) running here, just tabix (actually bcftools which uses tabix) to select variants in a particular region, wrap them in JSON, and return to the client.
Setting this up on S3 requires configuring CORS... the igv browser also uses tabix indexes and provides guidance on how to set this up [2]
[1] https://www.htslib.org/doc/tabix.html [2] https://github.com/igvteam/igv.js/wiki/Data-Server-Requireme...
I created a similar solution to what you have done using this alternative approach, writing a wrapper in R that invoked bcftools under the hood. The dataset I was working with was a lot smaller (1.6M Snps x 252 individuals), but should work with larger genotype sets as well.
I think nim is the best thing out there for building tiny CLI utilities. It’s a fun language to code in.
See also:
https://www.businessinsider.com/us-mint-ends-the-dollar-coin...
function pman() {
man -t ${1} | open -f -a /Applications/Preview.app
}Who am I to feel so important that I should take up space in perpetuity?
After a few generations, document the gravestones and get rid of them and make room for a park you can run around in.
Doesn’t this assume that code as written was of the highest quality? Under tight deadlines or misaligned priorities code quality suffers. Even if it works or “is right” doesn’t mean it wasn’t done to a standard that maximizes productivity.