Just my 2 cents of Mongo hate :-)
the engineering skill here is the ability to trade off risk vs benefit... I will tell you from my own experience the best designed software systems I have personally dealt with tend to use components somewhat behind the curve.
For me the best software systems are those that are well architected and use the best available technology. This doesn't mean we all should be writing Tomcat, Oracle, Apache stacks just because they are less shiny.
As a small startup, you might not have the time for extensive testing and patching of new hot technologies, which facebook, twitter, netflix do.
And not sure if you've worked for a large company but they largely comprise lots of little startups sized teams. The same principles apply regardless e.g. spiking technologies out, managing risk etc.
What I have an issue with is these stupid generalisations. Less shiny = good, Shiny = bad. The merits of the architecture and technology seemed to be completely ignored.
Which then translates to buggier software as you add more and more new features.
There is a reason we rewrite codebases, no ?
With less shiny I mean that it's old. Not that it is crummy qualitywise. Old code is not like wine. If it was crap then it will be crap now. Old code is like an old house. If it's well made and tended to, and built on solid principles it can last generations.
Extensible domain logic is something that generally does not age well. Old utility libraries with well defined interfaces, on the other hand, are invaluable in technical computing.
If you don't run at the same scale as those guys you won't hit those same limits. But if you do reach the scale of those guy's you will find that your needs are suddenly very much a unique snowflake that will require you either creating something new or heavily tweaking something that already exists.
Plenty of time for that when you reach the scale that justifies it though.
The point is, if you are going to bet your business on a technology, it helps if it has been tested with production workloads in many different conditions and scales. You want to know about as many shortcomings as you can. For many of the use cases that people use things like Cassandra for, they will be tolerant of 30ms++ reads and potential read inversions. Redis is used pretty heavily, but it is relatively simple code and you can trace through the entire writepath pretty easily and get a sense of its limitations (being single threaded is a blessing and a curse, you really need to be careful about bad tenants because a single slow query will cause an availability event for everyone). HBase is used in a few places, but usually only for cold data after they expect it to be read only occasionally and they don't want to use up space on their MySQL pci flash devices for it anymore. There are a bunch more, but they all have some latency, consistency, or availability downsides compared to a traditional sharded B+ tree backed transactional store.
No software system is perfect, but there are definitely practical balances to be made. Especially when you are beyond what a single database/server can offer in terms of write throughput. The fact is, when your traffic needs exceed what a single database can keep up with in terms of writes, you have to give up some level of reliability.
Granted that you can only prove that the system is vulnerable and not the reverse, but if there is a vulnerability it is much harder to trigger it.
Postgres is a good DB, but since it's not distributed, it's not very useful to compare it to distributed databases. Yes it's consistent, but it's only as reliable as the single node where it is installed.
[1] http://www.datastax.com/wp-content/themes/datastax-2014-08/f...
All of that said, you have to take a research paper funded by a database company (Datastax is backing Cassandra) with a grain of salt. Not to mention, that most people reach for MongoDB because it has some flexibility, and is a natural fit for many programming models. Beyond this, setting up a replica set with MongoDB was far easier than with any other database I've had to do the same with... Though I'd say getting setup with RethinkDB is nicer, but there's no automated failover option yet.
They also were quite generous by comparing load using non-durable write for CouchDB, HBase and MongoDB against Cassandra's durable write.
From my personal experience many scaling problems that you have with MongoDB once you switch even to a relational database that can't scale out are laughable.
Regarding Mongodb, all I'll say is that I've switched from mysql to mongodb 2 years ago, and I've never looked back. YMMV.
I'm also a user of ElasticSearch and Redis, and looking to add Couchbase to the lot. One size doesn't fit all. mysql and postgres certainly don't fit all either.
Also saying that Postgres cant scale horizontally is not entirely true, it in fact can[2], it is currently more complicated but I learned something when I was investigating how our applications would behave with Postgres backend. Turns out that every instance we had mongo we could run postgres on a much smaller instance. In one instance the data was so laughable that you could just run postgres on the same node that was running the app.
The point of it is that even if you think that you need to scale out, unless you're Google, Facebook or similar company you chances are you don't.
[1] http://www.enterprisedb.com/postgres-plus-edb-blog/marc-lins...
I suspect he did not go over replication, because Postgres technically still fail over support is DIY, although he should. There are two replication methods though which I would like to see:
- asynchronous - this one is fast, but it most likely would have similar issues the other database have - synchronous - the master makes sure data is replicated before returning to the user this should in theory always consistent
You would typically have two nodes in same location replicating synchronously and use asynchronous replication to different data centers. On a failure, you simply fail over to another synchronously replicating server.
Regarding consul/etcd actually those technologies did not do well in his tests, but authors appear to be motivated to fix issues.
That's why I said it's unfair to say "postgresql did well".
Call me maybe is supposed to test all the difficult problems of CAP, which have not been tested at all with Postgresql.
Can anyone explain (SQL pun not intended) to me the advantages / disadvantages between rethinkDB and say PostgreSQL?
A lot of the things would also apply to PostgreSQL
Almost every single IT company exaggerates claims, says their product is "the best" and "amazing" and suitable for every use case under the sun. Oracle did it. Microsoft did it. Mongo did it. And thousands of companies in the future will do it in the future. It's called Marketing.
And I think you should speak to all these customers (http://www.mongodb.com/who-uses-mongodb) and tell them they don't serve any real world need. I would imagine a few would ask the same question of you.
The day we start jailing people for their marketing hype is... well, it wouldn't be a good day.
And I can't imagine any of this is relevant today given that MongoDB allows for pluggable storage engines.
I always find it amusing when people bring these issues up because it's like a giant sticker on their forehead that says "I've never actually seriously spent time with MongoDB before". I always go through the configuration of the database I use to make sure it meets my needs. Only seems sensible.
Thats dubious. 1.) When MongoDB was released, none of the drivers used the "safer" settings. 2.) 10gen, at the time, released benchmarks with the "unsafe" settings comparing it to MySQL and boasted that MongoDB was much faster (ignoring the fact that it wasn't acknowledging your writes).
AFAIK, until recently (i.e. the last month), there weren't any such benchmarks released by MongoDB - and then, only for 3.x.
I'd be very surprised if any such benchmarks exist, as you claim.
Disclaimer - I work for MongoDB Inc.
Although to be fair, it's not just MongoDB that is performing poorly.
And it's pretty cool that you can make a choice between fast writes or safe writes. You can't have both but at least you can have the choice.
Having said all that, I generally prefer Postgres in almost every possible case.
However, this RethinkDB project looks sexy and with a great potential.