ACID gives you a lot of nice guarantees and it's silly to give it up if you have the choice.
As for MongoDB in particular, I would never use it for anything. It doesn't seem to do anything particularly well. There are better specialist 'NoSQL' databases that have actual benefits to fit your problem.
https://www.rethinkdb.com/docs/sql-to-reql/javascript/
Frankly, I cannot understand to write queries that verbose and complex when SQL looks clear and concise.
If I'm missing on how good RethinkDB is, I will gladly like to know.
One part SQL is notorious for is you don't know where it could fail until you send the whole query and receive the error unlike these NoSQL where operations can happen progressively.
While ORM can sort of mimic that behavior, I don't like that it hides SQL which makes crappy SQL alot easier to write and harder to debug.
There are niches though. Graph databases, search indexes, databases for logging, time series databases, mapreduce, etc.
hell, you can even use your favorite (postgresql) rdbms to queue tasks
Most of the real use cases for NoSQL involve systems where writes dominate. These are data collection systems where, later, someone searches the data. Surveillance, geophysical data, that sort of thing.
Now, there are many use cases for non relational data stores. There are already many answers in the comments but first you have to ask yourself what you need the data store for. In the cars analogy it's where you need to go, how fast, carrying what, etc. Do you need a glorified hash table? Redis is good at that. Do you need very fast queries? You might try Cassandra but be very careful at defining in advance all the queries you're going to run: if you need sorted results you have to plan the columns for them. See http://www.planetcassandra.org/blog/we-shall-have-order/ and remember that NoSQL doesn't always mean schemaless.
Sometimes the choice is about the tradeoffs between the ease of use, installation and planning. MongoDB maybe doesn't particularly excel at anything but you install it with an apt-get (check in advance which version you get) and adding replication is easy. However you better start planning at least a partial schema for your documents quickly or you'll get ton of litter in your database. On the cons side, its native JS based query language is awfully verbose and complex compared to almost anything else, ORMs and especially SQL. Luckily we don't need the console much.
To recap: ask yourself which data you have to manage and why, then google for a database that optimizes your use case and check it against the posts at https://aphyr.com/tags/jepsen which does a super-excellent job at finding out how dbs fall apart in extreme but realistic cases. Often you'll find that a relational database is good enough. Other times you'll have to choose between relational and one or two categories of NoSQL. There are very few mainstream RDBMs but there are zillions of NoSQLs so the choice might not be obvious. Final car analogy: car + tent or camper van?
The real question is:
In what areas are (recently build in the internet era) NoSQL Databases beneficial over (barely-)Relational traditional RDBMS Databases (build decades ago)?
----
Scalability, the easy to mutate schemas and all that are artifacts of our times.
MySql, Sql Server, etc are barely relational databases, coupled with a almost-decent way to interface them (sql).
For example, in them you can't store a relation inside a field. Schema manipulation is hard. A lot of potential programming power is not possible, without convoluted recent additions the Sql language, or hacking together strings.
And most of them have been made for workloads and scenarios that are at odds to what internet-scale companies need.
Unfortunately the solutions was do "Nosql":
"Any sufficiently complicated database management system contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of a good RDBMS."
---
Check the relational model. Is far simpler than people give credit, even without get too crazy like some purist want.
In fact, I don't see any reason why the relational model can't be used as flexible how json is applied in the backend.
---
Despite all the above and more, a traditional RDBMS is solid enough that most people not need to get crazy and move to (full) NoSql. However, could be good to use them in specific cases (like caches, search engines, etc).
But I have experience people using NoSql stores and trying to do the kind of work a RDBMS have solved easier and faster.
The cost: transactions, relations. These are actually incredibly valuable things to have.
If you're storing billions and trillions of records that don't need relations, NoSQL is great. You'll keep your fast lookups and saves no matter how big things get.
This can matter a lot more if you're a start up with the goal of having trillions+ records some day. Your SQLServer will eventually not be able to scale any higher, and then what? Moving to Nonrelational from relational is so painful.
Transitioning database is painful, but dealing with the extra complexity of non-ACID stores can inhibit your growth. If you reach the scale where you need to change, you'll probably be able to afford more developers to help you.
I call shenanigans on that - Google's experience was that they tried it and then got bogged down as every developer had to roll their own mechanism for waiting for syncs - hence they developed F1 instead...
This is quite arguable. Before the Computer Era, all businesses, including the whole financial sector did just fine without ACID. Still, ACID is used only at a local scope. I can often see ACID requirement used rather as an excuse for poor data modeling than a real need.
Can you elaborate a bit? I believe that in the pre-ACID world, business processes were much slower, not online processes like today. When you're only really changing data once a day, backups and manual intervention are acceptable options.
> Still, ACID is used only at a local scope.
What do you mean by "local scope"?
As for "local scope" I meant "at a single business entity". If you look at a banking or online payment systems as a whole then they are not ACID, they are eventually consistent. The are based on two basic principles:
1. write everything down at least two times, so you never lose data
2. updates are incremental, so you never overwrite data
Of course when you want stuff to happen at a faster-than-human timescale - or when you just want lots of stuff to happen - an ACID model might not be the right fit.
> As for "local scope" I meant "at a single business entity".
Understood.
http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on...
http://www.enterpriseintegrationpatterns.com/docs/IEEE_Softw...
The standard of the time looks quite ACID-y to me:
https://en.wikipedia.org/wiki/Double-entry_bookkeeping_syste...
They're approach is to use a Key-Value DB to build a linear-scaling PostgreSQL-like Database.
Personally, I'm just waiting for that to become a bit more stable, but it's probably the most promising in this regard and already very usable.
If you come from an RDBMS, you're probably going to dislike MongoDB. I like it for prototyping/MVP but once the schema settles down it's time to move.
However, NoSQL does not start and end at MongoDB, no way. There are lots of different flavours, from huuuuuuuuge key/value stores, to timeseries, to write-one-read-many document stores, to remote syncing mobile app datastores and probably loads of other types that I haven't used yet.
So figure out why you want to use NoSQL (CV building can be a reason too) and play around with some toy installations.
One example from my work is ingesting usage information from a variety of products, like a system that takes in information about all your utilities - power, water, gas, and so on. Each product will have its own attributes that's important to describing it. In a relational world, you might: add columns to your table as new attributes need to be described; come up with some kind of composite attribute system where you try to encode multiple pieces of information in one field; go for a star schema so every product gets its own table; or pivot so that each "record" is represented by multiple rows and the attribute's identifier is a field. Each of those approaches has their ups and downs but they're all present due to hard schemas. In a NoSQL world, a single table can have wildly varying records, each of which is useful on its own without needing to spread metadata into other tables.
- Bootstrap projects with open or changing requirements. A relational model is usually harder to maintain.
- Storing and querying objects with complex structures, such as those received from APIs.
- Having a single programming language across your entire application stack (ie MEAN). Sometimes comes up as a "should" or even a "must" requirement.
- Fast and simple object storage (ie sessions).
> - Fast and simple object storage (ie sessions).
Those would also be fulfilled by using a jsonb column in Postgres, right?
For querying, my guess is that frameworks such as MongoDB's Aggregation (https://docs.mongodb.com/manual/aggregation) are more versatile.
If you have a table with a column data in format json or jsonb, you can do a
SELECT data->>cust_id AS cust_id, SUM(data->>amount)
WHERE data->>status = 'A'
GROUP BY data->>cust_id;
Or, for the third example,
SELECT DISTINCT data->>cust_id;
Storing JSON is now natively supported in MySQL 5.7
Look at my answer to the sibling comment for an example.
Honestly, many (most?) NoSQL databases also don't really do them well. If you need these two properties, the only valid choice is really a master-less, shared-nothing database system. E.g. Apache Cassandra.
I know this is not a good example compared to a production issue, however I think the reducing the amount of time need to get started with a project is very useful over something that needs a lot of configuring before it can be used.
It's very easy to fall into the trap of "everything is relational" or "I could do that with Postgres/Oracle/etc". There are a lot of problems that have good RDBMS centric solutions, but you don't have to look too far to find end users or developers who are unhappy with the RDBMS solutions that they work with on a daily basis.
At some point you'll probably have to deal with schema anyway. If this comes at a later stage, you'll have to deal with a lot of diversity and inconsistency on potentially many points. This can become much more painful and time consuming than having dealt with it from the beginning.
If you're thinking about NoSQL as something like a simpler, modern SQL alternative, you're probably having a faster start and a lot of problems in the long run.
Scalability is something, NoSQL databases have for them, definately.
Not necessarily, if you have a nice framework. Years ago, I was working on an older Perl application with a tiny self-built ORM (a single class of about 1000 LOC). I gave it pluggable storage backend support and added a prototyping backend where the object is stored into a simple SQL-backed key-value store (a table with columns "id", "type", "data"; where "data" contains a JSON document). So when building something with a new data type, you could just start with the prototyping storage, and once your feature was working, switch to table-backed storage without much code changes.
That's one of the places where it made sense to use a NoSQL solution. Our documents had multiple levels of nesting, but no joins. We also use MySQL for our relational data, so in the end you have to pick and choose. All of the research we did pointed to NoSQL being a great fit for storing nested documents for stats collection. We haven't regretted following that advice yet.
SQL started getting big because it was the one size fits all approach that worked 90% of the time. In fact it got so big, that universities started to teach only them when they talked about databases in general.
But there are many problems, where SQL is not suited.
If you got hundrets of terrabyte of data you need to analyse, you're probably better of with a Hadoop cluster.
If you need inter process communication, Memcached or Redis is worth considering.
If easy db-sync is your main goal, couch/pouch-db is probably a good choice.
etc. pp.
But these three DBs alone are so different, that they can't be compared like MySQL and PostgreSQL can.
It's all about trade offs. If you're small just pick the best one for the job at hand. When you need scale you can usually hack either into scaling well enough.
Search versus query [for some definitions of 'search' and 'query'] seems like a case where business logic suggests an advantage of one approach versus the other.
Good luck.
speed of writes, there are no constraints that need to be checked with nosql (typically), it's like writing a value into a hash map. with sql there needs to be a check for multiple constraints: uniqueness, null, datatype, etc. if there's a relation then it needs to check those constraints as well.
the whole thing about rdb's is that they're trying to normalize data where possible, you deal with id's and the fields of the table can change without you needing to propagate these changes to other places. in nosql, if you have relational data then you have to manually make sure all these changes are propagated, pretty annoying if you ask me and quite error prone, you end up writing logic an rdb already optimizes for you. take "cascaded deletes", too, this is manual in nosql.
the best usage of nosql is if you have stagnant data, and you need to write it quickly, take a log file contents, for example, or if your data consistency requirement is not that critical (let's say you implement likes for a web post system, if you're missing 300 here or there, not a huge deal). other cases are things like counters, if you have a game where you're tracking scores, things are changing rapidly , but there's really not a lot of relation with other data, redis is a great example of this type of nosql storage, built in incrementing counters.
most of the time when someone brings up nosql for an application, my initial reaction is that it's a premature optimization, for a lot of data, there are tight relationships and rdb is great for that. but i tend to see that you need both in many cases.
End of the day, can't most things in the real universe be broken down into a schema? Be it DNA mapping, geological data, customer invoices, product listings etc.? Everything eventually can be contained in a discrete, defined schema?
I've dabbled on and off for several years but have yet to come across one situation where NoSQL would be a better solution than SQL. As others have pointed out - as soon as you start relating table together, NoSQL starts working against you.
I acknowledge that I have 30+ years with SQL, so obviously my viewpoint is biased by what I know. Could the hesitation to adopt SQL by most new developers point to the fact that they find it too intimidating and structured?
Full-text search. Just compare the speed of ElasticSearch against "SELECT * FROM logs WHERE message LIKE '%foo%';".
Also other niches: time-series databases, graph databases, etc.
At least until you reach several hundred gigabytes, postgres uses less resources for the same search speed.
This video shows searching through 3 GB of IRC logs, filtering, ranking them based on similarity to the query, creation time, likelyhood that it’s the type of message the user wants, highlighting the part of the message with the matching words, and displaying all that:
https://dl.kuschku.de/videos/2016-09-16_04-03-36.mp4
Postgres is, for smaller amounts of data, more than fast enough.
I'd recommend other NoSql databases over MongoDB like RethinkDB.
Also, SQL like databases support JSON. So Postgres and MySql 5.7.x I think supports it.
The only think I don't like about sql is that I have to know what my data looks like in advance. Set up is a pain
That can be good strategy when dealing with changing data formats:
1) Import the data "as is" into the database.
2) Process the data, extracting the data you need for querying.
3) Whenever parsing fails, figure out why and fix it (update your source code, throw away bad data, whatever), then rerun the 'process' step for that data.
Reparsing can be made reliable if you use transactions in the step 2. With a NoSQL solution, it may be hard to guarantee that you don't lose a few records ('may' depends on the specific NoSQL solution and the amount of manual work you are willing to do to restart your pipeline). (You can also postpone discovering your data problems, but that's delaying the inevitable)
What NoSQL is particularly good at is running on multiple machines. It achieves that by giving up some letters of ACID.
It also typically makes it easier to store and query unstructured data than SQL databases do, but SQL databases are catching up.
Compare this to an RDBMS-based web app, where to display a blog article for example you'd need to make multiple queries of different tables: first the "article" table to get the text, then the "author" table, then all the comments, tags, images, or whatever. The NoSQL way is much easier to code for this sort of thing. The lack of ACID doesn't matter so much because these kinds of data are usually written just once, read many times, edited rarely, so there are few opportunities for inconsistencies to creep in.
Haven't you heard of joins?
Where it becomes extremely impactful is when a database is distributed or sharded across a cluster. If the various tables (article, author, comment) are stored on different servers, it takes time for the database to reassemble them. Moreover, modern web apps often have content that's written once, read many times: blog posts, photo posts, status messages, news articles, advertisements, catalog listings, etc. Why re-do that work of assembling the content every time people want to look at it? When documents are rarely or never edited, the relational model's defenses against "anomalies" become less relevant. Far better to use a document store like MongoDB, which stores a piece of content as an integrated whole at a single place in the database.