Hacker News Reader

> it could be applied to tasks where Spark is normally used

How does it relate to Spark? Is it able to read from HDFS, translate to Spark map-reduce jobs or integrate somehow with Spark jobs?

View on HN

It could be a technological (column-oriented) alternative to what map-reduce in general and Spark with RDDs in particular are doing. It might be easier to define a data processing workflow as a number of column definitions rather than as a graph of collection transformations. And such a column transformation workflow could be also more efficient at run time.

Here is my implementation of this column-oriented approach as a Java library for server-side transformations: https://bitbucket.org/conceptoriented/dc-core

Reply on news.ycombinator.com