Hacker News Reader

Comment by drawkbox | Hacker News Reader

"They all have trade-offs. Key-value stores eliminate your ability to access data by anything other than its key. So, you have an article with an id of 5. What if you want to look up the articles by author 9? "

For cases like that you don't have one source of data, you do almost like an AFTER INSERT trigger to update other flattened tables of data that you need.

For instance, where you would normally perform a JOIN you would just after save update a JOIN table or a VIEW that is created by incremental updates.

So then you could filter by many other things and really aren't limited too much by space to make those duplications.

It happens with large RDBMS systems anyways after about 5GB of data, same type of scalability decisions have to be made.

But yes the article is off and you are right, most likely systems will approach this with multiple solutions for specific problems.

View on HN

> It happens with large RDBMS systems anyways after about 5GB of data, same type of scalability decisions have to be made.

I work every day with a > 4TB database and its all on 1 machine (a very big machine) with some very fast SAN. If you have money, Oracle can get very big on the right hardware. I am willing to bet MySQL or Postgres could do very well on that sort of hardware too.

True but scaling vertical is really a luxury, in most situations especially free to play or open systems for a certain level of usage, this is typically not economical. For a good vertical scaling strategy you need servers that reach into the tens or hundreds of thousands of $$.

The interesting thing about the article is that the NoSQL movement has succeeded in reaching outside the database nerd circles! To get some insight in the actual trade-offs and uses cases for non-RDBMS database systems, this article is a much better read: http://thinkvitamin.com/dev/should-you-go-beyond-relational-...

It happens with large RDBMS systems anyways after about 5GB of data, same type of scalability decisions have to be made.

More like 5Tb these days.

Well, the disk space is not the controlling parameter usually. Getting a ridiculously huge amount of fast, reliable disk is pretty much a solved problem. If all you were doing was writing logs at a constant rate then you could "scale" pretty much indefinitely just adding larger and larger disks to the same poky DB server, and it wouldn't matter if it was 5GB or 5PB.

The proper scaling issue is, of course, with the cpu and memory of your single DB server - and its disk speed - which can only be expanded so much, and at exponentially increasing cost. It's conceivable that even with a fixed amount of data, say 100G, you might have to confront scaling issues with increasing site usage, long before you reached anywhere near the limits of your raw disk space.

One database I work on was under 2Tb when I got it a little over 3 years ago and is nearly 30Tb now, so I am very familiar with the scaling issues. No need for sharding or any of that nonsense either!

That sharding is required once your DB exceeds what you have of physical RAM seems to be a irrevocable meme in the MySQL world.

Seeing the amount of people generalizing that to apply to all RDBMSs makes me think they don't have any DB-knowledge outside MySQL.

That's crazy advice even for MySQL. Having a DB that exceeds physical RAM is why you use a DBMS.

Who said that? Certainly not me, and I doubt gaius has 30TB of RAM so not him either. How much RAM you need is a complex question and revolves to a certain extent around the DB caching the "hot points" but differs so radically app by app there's really no rules or general guidance you can point to. It comes down to measurement of your particular installation, as always.

And fyi I prefer PostgreSQL. But MySQL isn't THAT bad, c'mon.

I'm not sayin MySQL is that bad, but I exclusively hear this kind of advice and anti-DB rhetoric from people running MySQL and MySQL only, who are complaining about scaling.

MySQL may be able to perform better, as I haven't done in depth tests I wouldn't know how far it can be stretched, but my point is that almost everyone complaining about the performance of the RDMS model seems to come from MySQL thinking that it represent the utmost limits of what relational databases can do, and that is quite sad.

Hm. Well, I agree with your observations. But maybe they're more willing to suggest alternatives because they haven't invested so much into learning an alternate system? You would expect someone who has, say, spent the last 15 years becoming an expert on DB2 to refrain - even subconciously - from questioning the very fundamentals his career is built on. MySQL users may not be advanced but at least they are unbiased.

MySQL is not that bad, no. With proper knowledge it can do a lot, and beyond its limits, capably configured, I am not sure any RDBMS can help. Facebook is not developing Cassandra because they can't afford Oracle.

I first used MySQL a decade ago... There are plenty of people who've fully invested their careers in it. Which is as sad as someone who thinks MS Access is the state of the art.

Sigh. Can't argue with that.

I should have clarified, I meant around 5GB per one type of data. Maybe a table or dataset that needs to be merged. For instance if you had a product database table that needed to be joined with another GB large or set of data at runtime.

At that point, even in RDBMS you have to stop with the joins and flatten. I have experienced these limits in Oracle and MSSQL around 20-30 million row tables.

Scalable fundamentals like database flattening, dimensional modeling etc are all what key value stores give you from the start. But really a good mix works best or project specific, just saying in the future with TB, PB of data the JOIN is a historic remnant.

It is possible one day that RDBMS will be seen as one of those evil optimizations that we made in our small relative worlds at the time.

I've tables much, much larger than that.

It is possible one day that RDBMS will be seen as one of those evil optimizations that we made in our small relative worlds at the time.

I doubt that for the simple reason that relational databases are maths: the relational algebra and relational calculus. There's no such theoretical underpinning to object databases.

I doubt that for the simple reason that relational databases are maths: the relational algebra and relational calculus. There's no such theoretical underpinning to object databases.

True, the relational architecture is moved to the object or code level, if you think about it in relation to size, it makes more sense there for the future.

The RDBMS was the brain but it is also the storage, code will now dictate how to use the storage and not the storage itself, the storage is becoming a component.

Reply on news.ycombinator.com