2,959 karma · joined August 22, 2011
Also, after opting out of personal data sharing:
> We're processing your request and should be done within 30 days.
Same for deleting your account.
>>> vehicles = hl.import_table('vehicles.csv', impute=True, delimiter=',', quote='"')
>>> t = vehicles.filter(vehicles.make == "Saab")
>>> t = t.order_by(t.year)
>>> t.show(3)
+-------+--------+-------+-------+----------------+-------------------+---------------------+-------+----------+-----------+-------+-------+
| id | make | model | year | class | trans | drive | cyl | displ | fuel | hwy | cty |
+-------+--------+-------+-------+----------------+-------------------+---------------------+-------+----------+-----------+-------+-------+
| int32 | str | str | int32 | str | str | str | int32 | float64 | str | int32 | int32 |
+-------+--------+-------+-------+----------------+-------------------+---------------------+-------+----------+-----------+-------+-------+
| 380 | "Saab" | "900" | 1985 | "Compact Cars" | "Automatic 3-spd" | "Front-Wheel Drive" | 4 | 2.00e+00 | "Regular" | 19 | 16 |
| 381 | "Saab" | "900" | 1985 | "Compact Cars" | "Automatic 3-spd" | "Front-Wheel Drive" | 4 | 2.00e+00 | "Regular" | 21 | 16 |
| 382 | "Saab" | "900" | 1985 | "Compact Cars" | "Manual 5-spd" | "Front-Wheel Drive" | 4 | 2.00e+00 | "Regular" | 23 | 17 |
+-------+--------+-------+-------+----------------+-------------------+---------------------+-------+----------+-----------+-------+-------+
showing top 3 rows
Hail's tables are functional. Operations like `filter` and `order_by` return new tables. That means it would be an error to use `vehicles.year` in the `order_by`, since the input and the sort expression refer to different tables. Unfortunately, this means you can't use `.` chaining.A little more background on the project: Hail's raison d'etre is a 3-dimensional generalization data frames we use for genetic data called a MatrixTable [3]. Conceptually, it is matrix-of-dicts rather than lists-of-dicts.
Genetic data is massive, so all of this is lazy and works on out of core data. The Python front end constructs an IR representing the query, it's fed through a query optimizer (written in Scala) and executed by a backend. We're working on multiple backends, but our primary backend right now is Spark.
[1] https://hail.is/docs/0.2/index.html
Harvard, the top biomedical institution, gets a score of 2,312.65. Alphabet (Calico isn't broken out), the #18th corporate institution, gets a 13.98, basically below all the major pharma companies. It wouldn't make the list of top 200 institutions overall. I didn't look closely at methodology, but it looks like it's counting publications in nature journals but doesn't weight for impact, which would surely up weight more prestigious institutions. That said, Alphabet/Calico/Verily's primary goal isn't necessarily to publish.
disclosure: I work at The Broad Institute (#66 institution, ha)
Reading a bit closer, I see the author describes CI as a sanity check, "ensur[ing] the bare minimum" and doesn't consider deploying on every commit. Maybe 3-7m is more realistic then.
However, I'm slightly surprised by this definition of CI. According to Fowler [0], "Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time. ... The key test is that a business sponsor could request that the current development version of the software can be deployed into production at a moment's notice." So having CI gates on the development version that are weaker than the release tests would not seem to be continuous delivery according to his definition.
We're currently releasing on every commit and our CI build (which implements continuous delivery) takes about 15m.
Who has a 3-7m CI build here?
Real power moves are exercising real power: saying no and being able to back it up, expressing a dissenting opinion, solving problems and providing real value, brining in money, controlling budgets or headcount, threatening to quit because you can/will, leading by organizing people to get something done, articulating a framing for situations that cut through the BS, understand people's motivations and what's actually going on, etc. These moves "take up space," but don't confuse taking up space for power. People with actual power won't.
So get to those meetings on time.
Too bad search engines aren't using this to improve search results.
Resources for Getting Started with Distributed Systems https://caitiem.com/2017/09/07/getting-started-with-distribu...
Distributed systems theory for the distributed systems engineer http://www.the-paper-trail.org/post/2014-08-09-distributed-s...
Distributed Systems for Fun and Profit http://book.mixu.net/distsys/
Notes on the Theory of Distributed Systems http://cs-www.cs.yale.edu/homes/aspnes/classes/465/notes.pdf
I wonder how they're doing logistic regression. It doesn't have a closed form and normally solved with an iterative, multi-pass optimization. edit: Read more closely. They're using (non-stochastic) batch gradient descent.
Google has written a bit about their monorepo culture. Here, for example: https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...
I got fake Dove soap from Amazon.
The situation with Lattice parts was the same; they reverse engineered them.