Hacker News Reader

Comment by spikels | Hacker News Reader

This is exactly what you should expect in almost all cases because you are kinda comparing apples and oranges.

Hadoop/Hive was not focused on speed but proving it is even possible to do queries reliably on such large datasets. Once this was achieved it immediately became clear the long waits for Hadoop/Hive batch jobs to finish made it impractical for many uses. Presto, Impala, Drill, RedShift, etc were all designed Primarily to address this problem and be much, much faster than Hadoop/Hive so that the data could be queried interactively.

All these new projects/products are in a very active competition to find the best way or ways to do this. You should compare RedShift to these other projects rather than Hadoop if speed is an issue for you. Each has it's pluses and minuses depending on the situation.

View on HN

Wait, so RedShift (Drill, Impala, Presto) do the same job as Hadoop\Hive, only faster? You only mentioned they addressed the speed, but what's downside? At what cost they achieved their velocity?

Basically, Hive is incredibly inefficient. Hive works by taking apart the HiveQL query and turning it into a series of Hadoop MR steps. A complex hive query may have many such steps, and for each one data must be read from HDFS, processed, and written back to HDFS. For most queries, the vast majority of your time will be spent just doing IO. Hadoop is also very, very slow to start up tasks (> 1 minute), so when you have a lot of them that can come to dominate your total run time.

Impala, Drill, etc. avoid all those unnecessary reads and writes by implementing the querying logic directly, rather than by compiling to Map Reduce.

Shark [0] is an interesting counterpoint. It takes essentially the same approach as Hive but on Spark instead of Hadoop, and achieves similar or better performance than the more "direct" implementations.

[0] http://shark.cs.berkeley.edu

The tradeoffs are both complex and evolving and are different for each project/product. In general Hadoop is more scalable and more flexible but much slower.

For example as of today RedShift can hold a maximum of 256 terrabytes of compressed data while Facebook's Hadoop cluster was over 200 Petabytes in late 2012. RedShift only supports limited query and data types and a single index while Hadoop can theoretically handle arbitrary data processing. But if these constraints are acceptable then RedShift will likely be orders of magnitude faster in most cases.

Other projects/products will have different tradeoffs but they are almost always faster as this was almost always the primary goal.

Where do you get the limit of 256TB compressed from?

     Amazon Redshift enables you to start with as little 
     as a single 2TB XL node and scale up all the way to
     a hundred 16TB 8XL nodes for 1.6PB of compressed user data.

from http://aws.amazon.com/redshift/features-and-benefits/

My mistake, thank for the correction.

The 16 node limit I was familiar with is not a hard limit. You can request more nodes.

http://aws.amazon.com/redshift/faqs/#0080

It would be interesting to see performance comparisons on these huge datasets. I would expect to see new and interesting problems at that scale.

Reply on news.ycombinator.com