Hacker News Reader

Comment by JunkDNA | Hacker News Reader

This is a very reasoned, well-written article. While I like the idea that the NoSQL movement is questioning some core database assumptions, I have often been uncomfortable with the meme that "SQL doesn't scale". Scalability is always a function of your circumstance. This article makes a good point that the group of people for which SQL doesn't scale need another option (internet startups with little cash and massive server load). While that case gets a lot of attention around here, it's important that people not try to extrapolate that experience to other areas of IT.

View on HN

What the NoSQL "movement" is doing is questioning some basic axioms of architecture and operations that may no longer hold in the modern era. There are many aspects to data, how it is used, how it should be stored & accessed, and what is or it not important within the set of available data. The RDBMS cabal set a particular set of standards (ACID) that were a good match for a lot of the early "big data" systems and the computing infrastructure of the time, but times change. As more people are bringing new data sets, new uses and access patterns for this data, and the computing infrastructure is changing from large servers to swarms of small systems the options for storing and accessing this data are also changing. I don't know many in the NoSQL crowd who are actually opposed to a RDBMS as a solution, but it is not necessarily the first choice for any problem; most of the people in this group seem to start by considering the data and the application and then picking and choosing the characteristics of the data system that will be used to meet these needs. The massive and highly distributed systems get a lot of attention (and they are almost always NoSQL systems) but that does not mean that alternative data systems do not have a place up and down the IT stack, nor does it mean that a RDBMS is not a good option for some situations.

"The RDBMS cabal set a particular set of standards (ACID)" What's the problem really? ACID or SQL? They are mutually exclusive. You can have ACID without SQL (BDB) or SQL without ACID (MySQL 3.X). Which of the two is the problem?

The RDBMS cabal set a particular set of standards (ACID) that were a good match for a lot of the early "big data" systems and the computing infrastructure of the time, but times change

Times change, but in the sense of adding additional systems with new requirements.

The existing systems have not gone away, and are not going anywhere. In fact there are more than ever of them around. I certainly still want my bank to run a system that has ACID transactions. Likewise my mobile phone's billing system, utilities bill, etc.

The real crux of the issue isn't that SQL won't scale, it's that it won't scale efficiently or effectively. SQL scales, but you need expensive, exotic hardware (InfiniBand switches, SAN storage) that requires niche engineering skills just to get up and going. By the time a startup-type company purchases all of this hardware and has figured out how to use it, the window of opportunity will have closed. In addition, you're then pretty much locked into support from the vendor for performance issues. With solutions like Cassandra, you've got the source and can solve your own problems, which, to a company rich in software engineering talent, is exactly what they want.

Even if you look past all that, RAC and ASE will run into limits as they're still relational, which is not a truly scalable approach to data modeling. There are real limits to their scalability as well. Oracle RAC will only scale to 100 nodes. That's a lot of dough for a 100 node limit.

So there it is. SQL doesn't scale (effectively or efficiently).

"So there it is. SQL doesn't scale (effectively or efficiently)"

I think you just re-iterated the author's reason for writing perfectly.

Instead of saying 'SQL' (the query language) doesn't scale, perhaps you can say full ACID doesn't scale cheaply.

We spent a bunch of time looking at large scale distribution at Berkeley for the Mariposa project. The short version: if you want to break ACID then you have to take the semantics of the data into consideration. Transactional financial data needs full ACID, many other types of data have looser needs (status updates, etc) and those looser needs can be taken advantage of.

As it turns out, even common DB's like Oracle, Postgres and MySQL can be used differently to get much cheaper scale for certain types of data. See the FriendFeed article on MySQL as a docdb (also the Friendly ruby mapping on github).

To paraphrase the author:

"ACID scales just fine if you need it (and if you need it you most likely have budget for it). If you don't need full ACID, there are plenty of other cost effective solutions you can run on low bandwidth disks/memory limited CPUs. Mix and match to fit your needs.

Please, enough with the religion."

You're arguing semantics. NoSQL = No"ACID-Compliant Relational Database with Normalized Table Design"

Uh, that is a bit of a leap...

It seems like every NEXT BIG THING comes in two parts.

1) The provocation: "Throw away design and do everything with testing" - "never use SQL again, just key-values" "Objects are Worthless!"

2) The sensible fall-back position you can use when challenged "Test are useful" - "You might not need full ACID", "We can benefit from shorter inheritance trees"...

I suppose every new idea needs a way to make a splash...

This is what as the triad of thesis, antithesis and synthesis. A lot of progress works this way. [http://en.wikipedia.org/wiki/Thesis,_antithesis,_synthesis]

Thesis (established view): SQL/ACID solves all you data problems

Antithesis: Nobody needs SQL/ACID, let's throw it all out

Synthesis: By carefully considering data integrity constraints we can find a more optimal data management solution for a particular problem (alternatively: let's go to the pub!)

Excellent plan. Alcohol always helps people get over arguments.

Bones heal faster than internet battles :)

> So there it is. SQL doesn't scale (effectively or efficiently).

You might just want to say "cheaply".

This is simply not true. I work for a major internet company that has millions of users. We use SQL for all of our data (images and SWFs excepted, of course). We do all logging to SQL tables. We use key-value cache servers for hot, non-relational data, but we don't have a lot of that.

We built most of our data center with cheap hardware. It is quite possible that we splurged on a load balancer and some other funky networking gadgets; I don't know. We did, however build quite a lot of custom software to abstract away the fact that we have split our data among dozens of machines. This is essentially the work of two people (one hardware, one software). Finding the people who can build such a system is difficult, to put it mildly, but once you find them, you don't need deep pockets.

Reddit is another example of a site that runs on SQL. Part of the problem with the "SQL won't scale" argument is that essentially all of the live queries function like key-value lookups (or lookups against a very small number of rows). If you're tempted to do something insane like service a request with a join, then you're doing it wrong and you need to cache.

Lookups by index will have the same performance characteristics no matter what. It doesn't matter if you're using a Python dict, an index in MySQL, or a key-value store.

THEY ARE ALL O(1).

Reply on news.ycombinator.com