443 karma · joined February 8, 2011
daniel.zidan.king@gmail.com
Also neat to see how bioinformatics is such a splitbrained community. They land next to R but are filled with Python projects.
Essentially, it’s easier to make tons of exceptions than to fix the root cause.
Are any of these three known for their skill at orchestrating teams or organizations? Compare to some of the more famous scientists, engineers, etc., they’ve led large organizations. Like, Jeff Dean, Steve Jobs, J. Robert Oppenheimer, etc.
The other posters are not wrong about compensation. Total compensation is off by a factor of two to three.
However, it is absolutely possible to work with a group of top-notch engineers on serious distributed systems & compilers in service of an excellent scientific-user experience. I know because I do. We are lucky to have a PI who respects and hires a diversity of expertise within his lab.
I enjoy being deeply embedded with our users. I do not have to guess what they need or want because I help them do it every day.
I also enjoy enmeshing engineering with statistics, mathematics, and biology. Work is more interesting when so many disciplines conspire towards the end of improved human health.
Moreover, I think we have differing definitions of “experiment”. In the context of a sequencing study, I think an “experiment” can be as simple as answering the hypothesis: does the missingness of a genotype correlate with any sample metadata (e.g. sequencing platform). You might try to test that hypothesis by looking at a PC1-PC2 plot with points colored by sequencing platform where the PCA is conducted on the 0/1 indicator matrix of missingness.
In the dry lab, that is what I mean by experiment. By that definition, a scientist does many experiments a day. Particularly for sequencing studies, these experiments are data-intensive, I need to run a simple computation on a lot of data to confirm the hypothesis.
Interactive analysis of large datasets (e.g. genome & exome sequencing studies with 100s of 1000s of samples) is well suited to low-latency, server-less, & horizontally scalable systems (like Dremel/BigQuery, or Hail [1], which we build and is inspired by Dremel, among other systems). The load profile is unpredictable because after a scientist runs an analysis they need an unpredictable amount of time to think about their next step.
As for productionized workflows, if we redesign the tools used within these workflows to directly read and write data to cloud storage as well as to tolerate VM-preemption, then we can exploit the ~1/5 cost of preemptible/spot instances.
One last point: for the subset of scientific computing I highlighted above, speed is key. I want the scientist to stay in a flow state, receiving feedback from their experiments as fast as possible, ideally within 300 ms. The only way to achieve that on huge datasets is through rapid and substantial scale-out followed by equally rapid and substantial scale-in (to control cost).
[1] https://hail.is
https://news.ycombinator.com/item?id=11244103
It’s about data structures for editable text. Surprisingly complex!
If you’re stuck at sea with a hurricane barreling down you can try to sail around the equatorial edge of it (in the northern hemisphere you try to sail south of it). Hurricanes tend to veer away from the equator. Moreover the wind and waves will be behind you, so you’re less likely to get knocked over by a gust or a wave. The boat is quite literally surfing.
If things get really bad you might heave-to which is a way to work the wind against itself causing the boat to mostly stall. It’s supposed to be very safe in heavy winds, but you would be pointing at the waves which is bound to be unpleasant.
Finally, a sailboat’s keel is very heavy. Check out the diagrams at [1]. The mast has to be well below water before the sailboat prefers turtling to upright.
Sphinx is a persistent thorn in our side, but we've made it work. A couple things I had to do to get this working:
- something thinks a filename with dots (e.g. example.8bits.bgen) means there is a python module that needs to be imported (e.g. example), and fails when that does not exist; we have files like this in a data directory
- I could not quickly figure out how to ignore files, so I had to delete a Sphinx conf.py file.
- Sphinx resolves the _templates folder relative to where it is called, not relative to the doc string's source file, so I had to add several symlinks (rather than manually edit every doc string).
---
OK docs are generating. Quite slow, maybe a couple minutes. Looks like one core is at 100% during this process. We've got ~400 files. Maybe its parsing some data files? The whole directory is 318 MB.
---
Now I'm looking at the docs.
First thing I notice is that Python type hint forward references break everything. Consider:
class GroupedMatrixTable(
parent: 'MatrixTable',
...
All my python function def's seem to be missing the function name?
No arg functions look really weird
def (
)
---
On the bright side, the mobile version looks great and the search works better than my experience with Sphinx.
EDIT: formatting
https://twitter.com/jeanqasaur/status/1074526838901202944
> Been having many conversations with people in their 20s about the paralysis that can arise from having too many life choices. My advice: commit to something and commit hard. Doesn't matter if you switch later. It's easier to prove yourself if you've had to do it once before.
> The quarter-life crisis can go on for quite a while--and you don't want to come out having little to show for your self-exploration. You learn just as much, if not more, about whether you like something by actually doing it, rather than thinking about if you might like it.
and then the somewhat morose conclusion:
> At one point in my mid-twenties, a friend observed that we were nearing the end of getting opportunities because of our potential and would soon be evaluated on what we've actually done. This was a real turning point in my thinking on how much space I should take to find myself.
The Broad Institute of MIT and Harvard was launched in 2004 to improve human health by using genomics to advance our understanding of the biology and treatment of human disease, and to help lay the groundwork for a new generation of therapies.
The Hail team's mission is to build tools to enable rapid analysis and exploration of massive genetic datasets (10s of TB and tripling yearly). We are committed to open science and everything we do is open source. We currently develop in Scala, Spark, Python and C/C++ but will use any tools we need to get the job done.
We're looking for skilled engineers who have a solid CS/engineering background, can quickly write clear, correct code and, for the senior position, have experience working on large, complex projects. You don't need experience in biology or our particular technologies. We work in a highly multi-disciplinary environment (with biologists, bioinformaticians, doctors, operations and mathematicians). Self-improvement is a fundamental part of our culture; we want to grow great engineers. You must be excited to be challenged and learn new things.
Questions? Want to apply? Visit broad.io/hailjob2017 !
Broad Institute of MIT and Harvard was launched in 2004 to improve human health by using genomics to advance our understanding of the biology and treatment of human disease, and to help lay the groundwork for a new generation of therapies.
Hail team is a small (5 person) software team building software to analyze and query large-scale (10s of TB and doubling yearly) genetic datasets. Our current technology stack includes: Scala, the JVM, AWS, GCP, Hadoop, Spark, and C. Our domain knowledge includes machine learning, bioinformatics, statistical genetics, compilers, and theoretical math. Hires need not have experience with every aspect of our technologies and domains.
Check out the official req at linkedin to learn more and apply: https://www.linkedin.com/jobs/view/316818823/
The example on the page is pretty straightforward. I encourage everyone to reason it out.
Others have noted the high rent/mortgage prices in the Cambridge/Somerville area. However, each couple could afford the rent on a one or two bedroom. We're privileged to be able to do so; however we choose not to.
Two of the four of us have experienced solitary (i.e. One bedroom, no roommates) living. We all prefer coming home to an occupied house with a pseudo-family.
For my partner and I, children are about a decade away due to the constraints of grad school, early career, and general desire to spend time together before adding a third or fourth person to the mix! Similar feelings lie with the other couple. As such, a traditional family is quite some time away for any of us.
Neither do any of us want to live with our parents for obvious reasons.
And living with just our partner feels, in both cases, a bit too isolated at this point in our life.
I suppose it comes down to the desire for family, togetherness, community. We each have three people on whom we can rely and with whom we live. It's comforting and enjoyable.
Of course we can also save significant money because we split one Internet bill, one rent, and one utility bill. Closer quarters living seems to be a bit more cost-efficient on the heating front. This compounds with the money saved by not owning a car and the economic benefits of easy access to the city's job market.
EDIT: I realize I didn't answer your question! I'm sorry.
It's common among my group of 24-28 year olds who are a mix of grad students and professionals. Most people have at least one non-partner roommate. I cannot think of one person who lives alone in the city.
I do have friends and relatives with apartments in the suburbs who live alone. They tend to have similar feelings to sibling posts here about "being an adult", a very strong desire to own property, and/or cannot stand the cleanliness issues that arise with roommates.
I've lived in various parts of Boston, Cambridge, and Somerville and we've previously had rock band and currently have room scale VR (it is, in fact, in use as I write this).
Neither had dedicated rooms but the setup and tear down is manageable. Neighbors never complains about the noise, but we also don't set the bass to room-shaking levels.
Perhaps the most significant difference between my home style in my last ten years and the decade previous (when I was a child/teen in suburbia) is that I share much more space. By pooling four incomes (two six figure tech salaries and two low five figure academia/architecture) we can afford places with enough living space to be comfortable.
It makes sense. When I was a child we at least had four people in our suburban house, but even then, it wasn't anywhere near capacity. With four adults in a four bedroom apartment we get much closer to capacity usage. We also make use of parks, coffee shops, and public libraries (CPL's main branch is beautiful) for more flexible space when necessary.
The first thing on that page are the computable real numbers. They have this annoying property that the set itself is not computable enumerable: there a real numbers that, if we only knew which ones they were, we could know all their digits; but there is no algorithm that will list all such numbers. https://en.m.wikipedia.org/wiki/Computable_number
There are portions of the academy that also care about constant factors. They would write what, I think, you mean by O(kN) as k * N.
If you need more precision, such as: "There's an order N term with constant factor k and some lower order terms whose constants I don't know (or don't care to calculate)", you could write that as: k * N + o(N). The little o means terms that grow strictly slower than N.
Edit: formatting
Edit2: I should have conceded that I'm not aware of anyone who attempts to use O-notation to talk about cache goodness. That certainly is swept under the covers of "Assume a perfect RAM machine" :P
> maybeT.map(t -> ...).orElse(...)
The author does make a good point that 'get' is unsafe. I think it was a necessary concession to team members who are learning how to program in this manner.
Perhaps what you're pointing out is that there are many dynamic and static ways to verify assertions about programs? Certainly.
When programmers talk about "type systems" I think they tend to mean a static, sound, computable approach to verifying assertions about programs.
JIF is a _type system_, albeit not a unification-based one like ML. Moreover, the situations the GP is discussing seem like things that don't need the powerful type system of JIF. Something like ML could handle those situations with ease.
I think contract often takes one of two meanings among non-researchers. Either people are referring to assertions that are dynamically checked or they're referring to very expressive assertions (regardless of when they're checked).
These are all different ways of proving statements about programs.
Turing-completeness is a not very interesting standard.
Iterating over lists, or integers in a range is pretty straightforward. Iterating over a tree? What does that even mean? I suppose we could take "iterate" to mean a catamorphism. Under that interpretation, structural recursion and "iteration" seem pretty equivalent.
Just feels like quite an extension of what "iteration" usually refers to in a language like C.
It seems like part of the benefits of the approaches described in the blog post is that the type system will catch incomplete extensions to the existing data or methods.
"Scheme was originally called 'Schemer', in the tradition of other Lisp-derived languages like Planner or Conniver."
Your reaction is, however, valid. People outside the Scheme community and it's closely related communities are unlikely to know this etymology.
I recognized it immediately because I also had to tear off the plastic casing. It shipped to me with a loose connection.
[1] http://www.amazon.com/Ultra-Slim-Bluetooth-Wireless-Keyboard...
There's also the wikipedia page on Substructural Types [2].