electrum

> For Linux and Mac (command-line only) I've gotten into the habit wrapping up a JAR in a shell script

There is a Maven plugin to do this:

https://github.com/brianm/really-executable-jars-maven-plugi...

We use it for things like the Presto CLI.

Symlinks work for gh-pages. We use it for linking to the current version of the docs:

https://github.com/facebook/presto/tree/gh-pages/docs

http://prestodb.io/docs/current/

Please note that warning in the manual that TRUNCATE is not MVCC safe. You might want to use a normal "DELETE FROM namesummary" instead.

This is a bit easier to read if you ignore whitespace (add ?ws=1 to the end):

https://github.com/boto/boto/commit/95939debc3813468264159d5...

System Preferences -> Keyboard -> Use all F1, ... as standard function keys

Oracle's documentation is excellent and is WAY more comprehensive than PostgreSQL. For example, most of PostgreSQL's SQL functions have a single line in a table to describe them, whereas each one in Oracle has it's own section with a full description, examples, etc.

I agree that the relative small size of PostgreSQL documentation and the fact that it fits in a single "book" compared to the many books that comprise the Oracle documentation can make it easier to find something. But can't say I've ever seen a product with as much quality documentation as Oracle.

You can find the documentation for Oracle 12c here: http://www.oracle.com/pls/db121/homepage

As a user, you want to look in "SQL Language Reference" first, followed by "Data Warehousing Guide" (found in a separate folder on the left side). As an administrator, "Concepts" is required reading followed by "Administrator's Guide".

Rails migrations do not support creating foreign keys in the database, but you can do it with this library: https://github.com/matthuhiggins/foreigner

This is a very common problem for buggy Java programs that have different versions of the same class on the classpath. The JVM will take the first instance of a class it finds on the classpath, so some orders of the classpath will work and others won't.

The solution is to eliminate any duplicate classes from the classpath and to enforce this in your build process. We use this Maven plugin to do it: https://github.com/ning/maven-duplicate-finder-plugin

The drag and drop works but it's a bit weird. You have to drag the tab into the other editor window, not on the tab bar.

No, it creates an additional editor area that can have any number of files open in it. You drag files between the areas or open the same file in multiple areas.

You can continue splitting any area vertically or horizontally to create a configuration that fits your working style or the code you're currently writing.

I replaced the mouse a long time ago with a Magic Trackpad and never looked back.

Facebook uses TAO for graphs: https://www.facebook.com/notes/facebook-engineering/tao-the-...

MySQL is used for storing textual user content like comments, etc. See https://www.facebook.com/MySQLatFacebook

Photos are stored using specialized systems: https://www.facebook.com/note.php?note_id=76191543919 http://www.stanford.edu/class/cs240/readings/haystack.pdf

And you can see some pictures of our new data center used for cold storage of older photos: http://readwrite.com/2013/10/16/facebook-prineville-cold-sto...

The benefits would be that you could query them using full ANSI SQL and could join them with other data sources such as Hive. One of the benefits of Presto's connector architecture is that it allows queries across multiple disparate data sources (after the someone writes the connectors).

Presto does read Hive data directly out of HDFS, bypassing Hive/MapReduce. However, neither HDFS nor the file formats like RCFile are efficient. The entire design of HDFS makes it difficult to build a real column store. Having a native store for Presto allows us to have tight control over the exact data format and placement, enabling 10-100x performance increases over the 10x that Presto already has over Hive.

Additionally, in the long term, we want to enable Presto to be a completely standalone system that is not dependent on HDFS or the Hive metastore, while enabling next-generation features such as full transaction support, writable snapshots, tiered storage, etc.

Wow, I didn't realize they had a VM image available. We haven't tried it, but there's no reason why it wouldn't work. Try it and let us know!

Facebook is always looking for interns and has a great intern program. See https://www.facebook.com/careers/department?dept=interns&req...

Please see this comment: https://news.ycombinator.com/item?id=6684785

Most of the data in our data warehouse is stored as relational records using Hive's RCFile format, though we are actively working on converting it to the new ORCFile format.

The Hive data model supports complex types including arrays, maps and structs (nested tables), and those are used liberally. We also have a fair amount of data stored as JSON inside of string columns in Hive tables, though much of this is actually structured and could be better modeled using Hive's complex types.

Presto currently has limited support for Hive complex types. All complex types are converted to JSON at query time and can be accessed using the JSON functions: http://prestodb.io/docs/current/functions/json.html

When running against Hive data, Presto is similar to many analytic databases in that queries typically perform a full table or partition scan, so "point lookups" that search for one or a few records will be much less efficient than they would be in an OLTP system like PostgreSQL that has a precomputed index for that query. (This is actually a property of the data source and not the Presto query engine. For example, we are writing an HBase connector that can take advantage of HBase's indexes for very efficient queries.)

In general, you should be able to write your query in the simplest and most readable way, and Presto should execute it efficiently. We already have the start of an advanced optimizer that supports equality inference, full predicate move-around, etc. This means that you don't need to write redundant predicates everywhere as is required with some query engines.

Also, if you are familiar with PostgreSQL, you should feel right at home using Presto. When making decisions for things not covered by ANSI SQL, the first thing we look at is "what does PostgreSQL do".

1) Presto always returns exact answers, with the exception of the approximate aggregate functions approx_distinct() and approx_percentile() (see http://prestodb.io/docs/current/functions/aggregate.html). We do not currently support COUNT(DISTINCT), though if you really need that you can use a subquery with SELECT DISTINCT.

We have a pull request for partitioned hash aggregations, but currently all the groups must fit in memory limit specified by the "task.max-memory" configuration parameter (see http://prestodb.io/docs/current/installation/deployment.html for details).

Regarding approximate queries, we are working with the author of BlinkDB (http://blinkdb.org/) to add it to Presto. BlinkDB allows very fast approximate queries with bounded errors (an important requirement for statisticians / data scientists).

2) Presto is a traditional SQL engine, so it would be ROLAP. We don't yet have any support for building cubes or other specialized structures (though full materialized view support with rewrite is on the roadmap).

The Presto query engine is actually agnostic to the data source. Data is queried via pluggable connectors. We don't currently have any caching ready for production.

There is a very alpha quality native store that we plan to use soon for query acceleration. The idea is you create a materialized view against a Hive table which loads the data into the native store. The view then gets used transparently when you query the Hive table, so the user doesn't have to rewrite their queries or even know about it. All they see is that their query is 100x faster. (All this code is there today in the open source release but needs work to productionize it.)

We have a dashboard system at Facebook that uses Presto. For large tables, users typically setup pipelines in Hive that run daily to compute summary tables, then write the dashboard queries against these summary tables. In the future, we would like to be able to handle all of this within Presto as materialized views.

3) We're excited about the Drill project. They have some interesting ideas about integrating unstructured data processing (like arbitrary JSON documents) with standard SQL. However, last I looked they were still in early development, whereas Presto is in production at Facebook and is usable today. Please also see this comment: https://news.ycombinator.com/item?id=6684785

Thanks! Presto supports arbitrary SQL including some basic window functions (full support is on our roadmap). I've used window functions at previous companies with commercial data warehouses to do some basic click stream analysis: http://prestodb.io/docs/current/functions/window.html

We have over a thousand employees in Facebook using it daily for anything you can imagine, so I recommend trying it and letting us know what you find useful and how it can be improved.

Hi, I work on Presto at Facebook and would be happy to answer any questions about it.

As the documentation for Random says: "Instances of java.util.Random are not cryptographically secure. Consider instead using java.security.SecureRandom to get a cryptographically secure pseudo-random number generator for use by security-sensitive applications."

GitHub really needs to implement this for Pull Request notifications.

If you use Java with Thrift, you should check out Swift:

https://github.com/facebook/swift

Swift is an annotation-based library for creating Thrift types and services. It was created to be an alternative to code generation, which is generally a pain for Java development. Swift generates bytecode at runtime, so you get all the performance advantages of code generation with none of the development disadvantages.

The idea with Swift is that you own the annotated classes and can fully customize them to your liking, independent of the IDL and wire format. For example, you can name the methods whatever you want, add documentation, helper methods, etc. When implementing a client for Thrift service, you only need the methods that you actually call.

You can use swift-generator-cli to generate the initial versions of the classes from existing Thrift IDL, which you then customize and maintain yourself. Alternatively, you can use swift-maven-plugin, which is pure Java (no external C++ binary), to generate code at compile time.

This project is in active development and use at Facebook, so please give it a try and let us know what you think!

The text is blurry on the Retina MBP in Chrome. Looks fine in Safari.

You use EBS and back it up using the built-in snapshot functionality, and/or back it up another way. RDS, which is presumably based on EBS, provides backups and snapshots for precisely this reason.

Amazon only claims "obscene number of nines" for S3 durability (99.999999999%). And this claim seems to be accurate: I've never seen a publicly reported case of anyone losing data. Anytime you read their forums about people reporting data loss, a typical response is AWS staff saying "we see delete requests for those objects on date X" with the users responding "oh, oops, we had this background delete process".

However, for EBS volumes, Amazon is very clear about the expected data loss rate:

"The durability of your volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot. As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume. This compares with commodity hard disks that will typically fail with an AFR of around 4%, making EBS volumes 10 times more reliable than typical commodity disk drives."

This problem seems to be fixed in recent versions of Oracle:

https://blogs.oracle.com/optimizer/entry/explain_adaptive_cu...

There are two books we recommend to every developer:

Effective Java: http://java.sun.com/docs/books/effective/

Java Concurrency in Practice: http://www.javapuzzlers.com/

Everything else is covered by the excellent online documentation, including the language and VM specification books: http://docs.oracle.com/javase/specs/

Java Puzzlers (http://www.javapuzzlers.com/) is a fun read that highlights many corner cases of the language and might be helpful for code reviews and debugging.