Hacker News Reader

Comment by deepstack | Hacker News Reader

SQL will always be faster than Hadoop and MapReduce. The main reason to use those other slower services is developer are not use to SQL or declarative programming, and insist on having the code in Procedural way.

View on HN

That's completely backwards. Mapreduce-like approaches are how SQL datastores are implemented underneath; the absolute best case for SQL is to equal hand-tuned mapreduce-like performance, and often it will be slower (you're at the mercy of your query planner to pick the right indices, do joins in the right order, etc.). The main reason people use SQL is because they find it easier to express a query that way (which is completely legitimate - if your query planner is good enough most of the time, you've got better things to be doing than hand-tuning your query execution).

No, that does not seem correct. SQL Datastores are not "map-reduce underneath", they have optimized datastructures for efficient querying (i.e. indices). Map-reduce is equivalent to those cases in SQL database where you have full table scan in your query plan - basically brute-forcing your way through the dataset.

You can (and often should) have indices in a map-reduce situation as well - you just build them in an explicit, visible way. But in most of the relevant use cases you're doing some kind of aggregation over the whole table, so indices don't help any.

And if your primary use-case is column-wise aggregation over the whole table, in SQL you'd use a (compressed) column store rather than a row store as your table storage method.

To be fair, Parquet, which is commonly used in Big Data solutions is a column store format. So, once you normalize your data and save it as Parquet you can have efficient column-wise aggregation - but that assumes some preprocessing step.

That makes no sense. SQL is a query language, commonly implemented by relational databases.

In the early 2000s, columnar relational data warehouses were not sophisticated and scalable enough to handle the scale of data encountered at Yahoo, Google and other internet companies. MapReduce (and the many evolutions of Hadoop ecosystem) was created to scale processing through low-level instructions and algorithms.

Eventually columnar data warehouses caught up and are now capable of handling petabyte scale, regardless of whatever language you use to query them. The fundamental storage and compute primitives haven't really changed that much, just offered in a much more user-friendly way now.

SQL itself is just a query language, it's the underlying cloud based data warehouse that fulfills the role of what map/reduce used to do in terms of parallelization transparently.

Reply on news.ycombinator.com