evancasey

ICYMI: cardiogram is doing this with the apple/android watch (possibly other devices). they're not using phone sensor data yet to my knowledge though...

The list of companies using Druid in production is impressive considering how recently this project emerged. We’re considering using Druid for our production analytics engine at Tapad (tapad.com). A significant portion of our ad-hoc and user-facing analytics applications involve large, multi-dimensional aggregates over ad impression/click streams, so Druid’s an obvious choice.

So far, mysterious failures when indexing have been major pain point. These seem to be especially problematic with high cardinality dimensions. Has anyone run into similar issues and/or other weird quirks in production? What about real-time avro ingestion?

Best of luck to the Imply team with this project. Whether we end up using it or not, I have no doubt that the Druid + Pivot stack will be a popular choice for folks building analytics products on large datasets.

Huge spark fan here. Love the execution model, API, supporting libs etc.

Unfortunately, Spark doesn't scale well on large datasets (10TB+). Sure, it's possible (and has been done), but right now there are too many rough edges to make it a better choice than Scalding/Cascading for data processing at scale. Most of this boils down to fine tuning certain Spark parameters, which is a pain when you're dealing with long-running, resource intensive workflows.

Does google maps/citymapper consume this sort of real-time train location data?

If not, perhaps you could use the NYC subway REST API to cross reference route calculations

Seems like a useful tool that would make doing data analysis on a remote machine much easier than with plain old AWS + anaconda.

However, the incremental improvement of renting an EC2 instance (even with 8 cores and 65gb) pales in comparison to using a distributed data processing approach with medium to large datasets. Any plans for supporting Hadoop/Spark + Amazon EMR in the future?

I did most of my benchmarking with the 10M MovieLens Dataset http://grouplens.org/datasets/movielens/ consisting of 10 million movie ratings on 10,000 movies from 72,000 users. So not necessarily "big data", but big enough to warrant a distributed approach.

Spark is ideally suited for iterative, multi-stage jobs. In theory, anything that requires doing multiple operations an a working dataset (i.e. graph processing, recommender systems, gradient descent) will do well on Spark due to the in-memory data caching model. This post explains some of the applications Spark is well-suited for: http://www.quora.com/Apache-Spark/What-are-use-cases-for-spa...

My experience using spark has also been nothing but positive. I recently built a similarity-based recommendation on Spark (https://github.com/evancasey/sparkler), and found it to be significantly faster than comparable implementations on Hadoop.

subprotocol's point about specifying the number of tasks/data partitions to use is true - you need to manually set this in order to get good results even on a small dataset. However, other than that, spark will give you good results pretty much out of the box. More advanced features such as broadcast objects, cache operations, and custom serializers will further optimize your application, but are not critical when first starting out as the author seems to believe.