Hacker News Reader

Comment by TheGuyWhoCodes | Hacker News Reader

Has it become easier to run ad hoc queries with spark? I remember a year ago that the only available solution was the job server by ooyala. Which seems to be a missing feature of core Spark, and isn't something I was willing to bet my product on.

Datastax evangelized people to use Spark to run queries over Cassandra but it looks so awkward and time consuming to copy jars around to the master, basically you need a dev ops team to this and even more scriptology for production.

View on HN

Infinitely. Yes. The addition of Dataframes in 1.3 and numerous enhancements to SparkSQL have made it as easy as:

  val myDF = sqlContext.read.format("com.databricks.spark.csv") //allows you to read a csv file (for simplicity)
    .option("header", "true")
    .option("delimiter", "\\t")
    .option("mode", "PERMISSIVE")
    .option("inferSchema", "true")
    .load(filename)
    .where(filter_query)
    .cache()
  
  myDF.registerTempTable("myDF")
  sqlContext.sql("SELECT COUNT(*) FROM myDF").show()
  sqlContext.sql("SELECT COUNT(*) FROM myDF where filter_query").show()

>to run queries over Cassandra

Why not just use Presto? It gives you basically full SQL capability for Cassandra with minimal effort.

Presto seems like a good product, haven't really test it too much tho. When we started to look at Cassandra 1.5 years ago, Spark integration was all the rage, and the premise it was the missing link for doing analytic on Cassandra.

We tested it thoroughly and came to the conclusion that is wasn't a mature enough solution.

Reply on news.ycombinator.com