Hacker News Reader

Comment by linuxhansl | Hacker News Reader

HBase looks bad in some of these benchmarks, because it is hard to setup and has many tuning knobs to be tuned correctly for the workload in question.

Due to its strictly consistent nature you have to think about key design, hotspotting of servers, etc, etc. In return you get correct atomic operations, row transactions, range scans by default (Cassandra uses a random partitioner by default not allowing range scans), etc, etc.

Some of the largest installations on this planet run on HBase. For example, FaceBooks HBase stats at HBaseCon (May 2012): Billions of msgs/day, 75Bn ops/day, 1.5M ops/sec peak. 250TB new data/mo and growing. (Facebook also created Cassandra, but is not using it)

As usual you use the right tool for the job and isolated benchmarks usually do not bear this out.

View on HN

And more usually, you'll use the tool the tool that your most familiar with or can become most familiar [quickest]. You say so yourself, HBase is relatively hard to quickly come to terms. We can observe the same when viewing the "SQL camp" with MySQL and PostgreSQL (and MSSQL).

It is also true of the Facebook HBase install! Completely unnecessary jab; the use of HBase was politically driven -- what the architects wanted to use. What are the proofs that it was the "right tool?" Well, it works, but you won't find it behind Google's Gmail / Talk, so who knows!?

Hmm... Where to start.

First, kindly point at an Cassandra installation, which supports the size that HBase supports in this setting.

Can I assume you have first hand information about this decision from someone at Facebook? I was stating a fact... not a jab.

Re: Google. Nothing at Google is driven by an eventually consistent store. They got that part right from beginning. It's too hard to manage from the an application point of view (unless you store immutable data, in which eventual consistency is pretty awesome). Checkout BigTable, Megastore, Percolator, Spanner. Except for the latter these are all based on BigTable. Note that BigTable is always consistent like HBase (and unlike Cassandra).

Are you saying the MySQL is better than PostgreSQL (or vice versa?) because one is easier to setup than the other. That is a terrible way to pick a technology.

Out of the box functioning is nice, easy to get started, makes the product look good in benchmarks, etc, but it just hides the complexity until you deploy into a production setting.

Lastly, I agree that if you store the odd few TB here or there, you certainly do not need to bother with the complexity of HBase.

Re: Facebook, as jbellis mentions in this comment thread, Facebook shards HBase, so it's not able to support what they need without some major additional complexity.

Re: eventual consistency. It's tunable to the need of the user. It's a trade-off. Tunable consistency becomes nice when you want a system to be always available. See the Amazon Dynamo paper, on which Cassandra is partially based.

I think many underestimate how many are using Cassandra in production. Spotify, eBay, Walmart, Netflix, Ooyala, NASA and others - one list is found here: http://www.datastax.com/cassandrausers.

Also they used an old version of HBase, 90.4 in the benchmark, HBase 94.0 has been out since May 2012, and has a lot of work to improve performance especially remove lock contention on the region server.

It's worth noting that the FB HBase install is also sharding across multiple sub-clusters because of the HDFS namenode SPOF problems [1].

Personally, if I'm going to shard manually I'll stick with postgresql. One of the primary reasons to use something like Cassandra is that it solves that for you.

[1] http://www.slideshare.net/brizzzdotcom/facebook-messages-hba...

My understanding of facebooks Pod Architecture for HBase was not the name node but simply scaling HBase, HBase gets rather unpleasant at facebook scales. The facebook HBase fork has things like compactions disabled to improve performance.

I ran a HBase cluster with 1PB storage, it became very unwieldy at this scale, thousands of regions and lots of tricks to keep it happy. As for SPOF, the name node now has HA and it works very well.

Interesting. Do you remember what kind of problems you ran into and what version of HBase you used?

HBase 90.4, problems with I/O we had a very heavy read load on top of a write load, write load bursting 14,000 TX per second, and an average of 8,000 per second - each record around 2k.

Because of the I/O the WAL had to be turned off, this introduced problems when Region Servers occasionally died. Implementation of large regions 10GB, and fairly large HBlocks 512MB, increasing flush sizes to reduce minor compactions. Use of MSLAB to virtually eliminate GC all together, use of large heap 12GB on RS.

Worst problems we experienced was META corruption, that really , really sucked.

Thanks. If there's a more detailed writeup you can point me to that'd be great. I would like to make sure then that all these issues are addressed in the current versions.

0.94+ has MSLAB by default, with HFileV2 (0.92+) we can support much larger regions (20G or bigger). Curious about the 512M HBlocks, did you have scan-heavy read-load?

14k TX peak per regionserver? x 2k that's 28M/s (56M with WAL). Should be doable now even with WAL (definitely with deferred flush). Well, maybe not with concurrent very heavy read load, depending on disk configuration.

Probably on top of Hadoop 0.20-append? Hadoop-2.x.x should be far better too.

It is also worth noting that Namenode isn't a spof in Hadoop 2. Namenode HA[1] has been in for a while and QJM[1] went in more recently.

[1] https://ccp.cloudera.com/display/CDH4DOC/Software+Configurat... [2] http://blog.cloudera.com/blog/2012/10/quorum-based-journalin...

Reply on news.ycombinator.com