AWS Redshift: How Amazon Changed The Game
blog.aggregateknowledge.com
blog.aggregateknowledge.com
I listened to the recent Redshift performance webinar and was surprised by:
- the hard maximum 15 concurrent queries (14 when you reserve one for management)
- the need to pre-assign queries into queues so that fast queries don't get blocked by slower ones
- the recommendation to vacuum after large inserts (showing its Postgres roots) and to limit query concurrency to 3 while vacuuming.
Have these limitations caused problems for anyone?
Edit: formatting, grammar.
What drawbacks have you found with Impala? I've been keeping an eye on it, and also Shark: http://shark.cs.berkeley.edu/
We haven't yet had a chance to optimize our environment either. For example, our nodes are still running a pretty old version of CentOS, so we have LLVM disabled (which would help a lot for huge batch computations...see http://blog.cloudera.com/blog/2013/02/inside-cloudera-impala...).
Also, our data is stored in RCFile, which is not exactly the most optimized columnar storage format. We're working on a plan to get everything over the new Parquet (http://parquet.io/) columnar format for another boost in performance.
We haven't come across any real drawbacks using Impala as of yet, it fits our needs pretty well.
Disclaimer: I work for Cloudera in their internal Tools Team, we like to dog food our stuff :).
Edit: One drawback of Impala is the lack of UDF support, but this is something that will be coming in a later release.
I don't work for them, but they make a good product.
Might be worth playing with what Bityota offers; Postgres is awesome but sometimes makes you bend over backwards to do things that are easier in other systems. Paraccel/Redshift has some good extensions, but sometimes there are better ways to do things.
(Seriously, what does it mean?)
Outside of Postgres, none of the truly open-source databases scale well with MPPs. You're still stuck with mostly single-core processing.
So you can pay good $$$$$ to the commercial database providers to provide you with a good database that may scale to your needs.
Or use Redshift for a great analytics DB at a reasonable price. (Note: we heavily use HBase where I am at, but that's comparing apples to oranges.)
* Disclosure: I'm an Apache HBase committer, I've written parts of Impala's HBase integration, and I work at Cloudera.
On the other hand, Impala, RedShift, Oracle Exadata, etc. let you ask the database to do work at a much higher level, which allows for much better performance because the data storage and computation layers can work in tandem (so you can prune down to only the data your query needs at each storage node before hitting the network, for example), and the database does the work of optimizing for multiple cores and nodes, not the writer of the analysis routine.
Well, except for the last such comment, which contradicted some of the prior ones by 1-2 orders of magnitude.
Although Impala isn't technically Hadoop, it's daemons are co-located with each your Hadoop Data Nodes, and talks directly to HDFS and HBase. Also, there is no ETL process with Impala, you could drop a few billion row flat files into Hadoop and query it within a few minutes.