Hacker News Reader

I agree with you - this article does a very poor job of communicating the advantages of 'NoSQL' databases over traditional solutions. It spends too many words setting up a straw man (SQL == not partitioned, not highly available, not redundant, etc) and too few actually making it's case.

Lines like this are, at best, a gross oversimplification of relational database performance tuning: "But when you need to optimize performance, you look at the queries you actually perform, then merge tables to create longer rows, and do away with joins wherever possible."

And this: "We require sub-second responses to queries."

Really? Single big-box OLTP systems have many issues, but if you are getting query times greater than a second on a typical online database then you are either doing something wrong or have specific requirements. Neither of those things will be fixed by blindly picking 'NoSQL' over 'SQL'.

"any significant database needs to be distributed." I could easily list, off the top of my head, 100 significant databases that aren't distributed. Maybe some of them should be, but saying that "any significant database needs to be distributed" is a real stretch.

> There are many good reasons to design around a NoSQL database instead of a relational one. This article provides fewer than zero of them.

Yeah, that's the thing. An article of a quarter of the length with more research, less strawman bashing and less breathlessness could have made the case for NoSQL much more effectively.

I also noticed the emphasis on analytics which is actually very much a weak point of NoSQL. You can't do ad-hoc analytics in a NoSQL database without performance at least an order of magnitude worse than youd get in a relational system. NoSQL only works with analytics when you know all the questions you want ot ask ahead of time.

"absolute consistency isn't a hard requirement for banks": see above. Yes it is.

This one gave me a laugh. Thanks for pointing it out.

Let me see.... Your business does nothing but manage money. You don't need absolute confidence over where that money is at any given point of time? Right..... In fact accounting systems (and by extension ERP systems) are about the LAST place you'd want to use anything other than a relational database system/

Re-reading the article I see what the author is saying, namely that some lag between the ATM network and the bank's accounting system is permissible. However, you still need to have a single point of truth that is absolutely consistent. If you don't, the bank's accountants (and probably federal regulators too!) will be rather unhappy! So the author takes a reasonable observation and twists it beyond recognition......

Most banks use an eventually consistent message oriented architecture, not a transactional one.

At a low level banks use both absolute consistency (for physical transaction stores) and eventual consistency (for logical transaction implementations). The logical transaction implementation abstracts inter-bank and physical payment messaging.

It would be impossible to have absolute consistency in the logical transaction layer as transaction scopes would have to be open (i.e. locked) for days at a time in some cases. That simply doesn't scale.

Ultimately banks have millions if not billions of pounds floating around not in traditional transactional stores all the time.

I agree that ATM's are a bad solution to this. The issue there though is loose coupling of third party financial networks though compared to a general need for consistency in terms of one's own financial needs.

But loose coupling between third party payment networks (say debit card purchases over Cirrus) and the bank is not really the same problem as using Cassandra at Facebook.

>"absolute consistency isn't a hard requirement for banks": see above. Yes it is.

There's a paragraph in the (long) article about absolutely local / eventually global consistency.

>"but they are hard to partition without extreme effort or extreme cost." ... Wh... Bu... That's not even wrong.

I don't know about Tandem but another comment mentioned Postgres-XC so I checked it out. According to the wiki its not globally consistent.[1] Care to elaborate on what isn't wrong, or name any (relational) alternatives to Postgres-XC?

"We need some research work for solutions on following issues.: Global constraint. Can we enforce unique or other constraint exclusion globally in multiple data nodes?" http://wiki.postgresql.org/wiki/Postgres-XC

I'm of two minds about consistency vs banks. On the one hand, yes, in the CAP sense, "consistency" is atomic consistency, and banks allow a form of eventual consistency. On the other hand, I'm not sure if the NoSQL sense of eventual consistency is strong enough for what banks need; how do NoSQL eventual-consistency models deal with conflicts that affect other tables? For instance, a quick Google shows that MongoDB offers "programmatic merge" - but would that also allow you to say "when merging my spouse's ATM transaction with mine in our checking account, also remove the spurious transaction in the bank's cash-on-hand account"?

Maybe I'm thinking too close to the metal; to get eventual consistency at a higher level, you still need atomic consistency at a lower level - even Paxos uses two-phase commit under the covers.

As for the partitioning, what "isn't wrong" is:

- He says the CAP theorem is historically irrelevant to relational databases, which is wrong, and that they have been "adapted" to provide high availability, which is kinda wrong, and then:

- He confuses "partition" in the "partition my database" sense (scale my database across multiple tables or nodes) with "partition" in the original CAP sense (do not permanently explode if a node goes offline), so he isn't even wrong - any multi-server database allows partitioning, by definition.

Also, I think you're misunderstanding that item on the wiki page; it sounds like Postgres-XC IS globally consistent, but that it can't yet support some important forms of consistency like UNIQUE constraints across nodes (you can only have a unique constraint within the same node). All nodes would see the duplicate rows, though, so it is globally consistent.

- availability means access to the data, not just uptime

- I have never seen changing schemas on the fly in a RDBMS programmatically (except admin functions). I have seen overloading of column types however.

- "super column" data stores have a deistinct design advantage that you can not get from relational DBS.

- everyone that is using relational does the same thing when they get big data: denormalize. From materilzed views al the way to sharding and replication.

- banks use transactions, but not as a two phase commit. it is a single atomic record - not partial updates.

- we still have some species from the age of the dinosaurs, but the land is rules by those that have evolved.

- I agree that banks should be consistent. Not sure what he author was saying here

- CAP is relevant to ALL datastores. Consistancy, Availability, Partition tolerance : pick any two.

- if your post does not get committed to Facebook, it may be important to YOU, but the application is designed more for availability

Of course you can change schemas on the fly. Just because ALTER TABLE goes ka-thunk on MySQL doesn't mean it can't be done in more robust databases.

I'm not sure what you think two-phase commits are, but they ARE atomic transactions. They are essentially "Everyone cool with this transaction?" "Yep!" "GO."

The CAP gotcha is that you already have partition tolerance - if you didn't, your database would go "boom" when a node went offline. So, really, it's "pick one".

And believe it or not, Facebook moved some stuff from Cassandra to HBase in part because of its stronger consistency model.

You can change schemas on the fly. But no one does it in their application except part of Admin.

Two-phased commits are when you write a partial record to the DB and then cleanup the transaction. When they complete succesfully, they are a complete transaction, but during a failure they are not atomic through the database because it takes extra application logic to cleanup.

Partition tolerance is not when one node goes offline, it is when critical nodes (master) or up to n/2 fail.

For CAP, you can optimize for just Consistancy, Availabilty, or Partition tolerance. But you can also pick two - you just can not do all three.

I did not know Facebook went to Hbase and why it do so - can you provide a link?

It looks like different people use different definitions of partition tolerance; the original meaning was:

"The network will be allowed to lose arbitrarily many messages sent from one node to another."

Which means that, if you have a network, you have partition tolerance, period.

But Stonebraker defines it differently:

"If there is a network failure that splits the processing nodes into two groups that cannot talk to each other, then the goal would be to allow processing to continue in both subgroups."

Great article here:

http://www.cloudera.com/blog/2010/04/cap-confusion-problems-...

As for two-phase commits, I'm not sure what system you've used that requires app knowledge, but (for instance) PostgreSQL does them automagically; you just do PREPARE TRANSACTION and COMMIT PREPARED, and if the transaction fails you need app-level error handling - but presumably you have that error handling even without two-phase commits.

Here's an article on Facebook's move to HBase:

http://www.facebook.com/note.php?note_id=454991608919