RethinkDB 2.0 is now production ready
rethinkdb.com
rethinkdb.com
How is RethinkDB licensed?
The RethinkDB server is licensed under the GNU Affero General Public License v3.0. The client drivers are licensed under the Apache License v2.0. http://rethinkdb.com/faq/
The cases in which the community forks a project licensed with a copyleft license (like LibreOffice) has to do with insatisfaction with the direction in which the company that owns the original trademarks is leading said project. There's no risk of closing the source code.
Slava @ RethinkDB here.
This has one exception -- the copyright owner can choose to start releasing enhancements as closed source, and they wouldn't be legally obligated to open source them. Currently RethinkDB, Inc. owns the rights to the code -- if we wanted to continue developing closed-source enhancements, technically we could. If we were acquired and our acquirer chose to do that, they could too (since they'd end up owning the rights to the code). Copyright and licensing are different things -- essentially the copyright owner has the right to relicense future code in a different way.
We have extremely strong incentives not to engage in bad behavior (and it runs against our beliefs), but I thought I'd point out that there is no legal barrier.
This is also why many open source startups don't accept code submissions from outsiders until they've determined they're going down the path of a consulting-focused business model. It's just too risky.
- You & and your team have put in 5 years of effort. - You have generously shared your mind product. The community has it.
I don't [know] about you, but over here in NYC, we have to pay for food, rent, clothing, medical insurance, entertainment, etc. Not a single one of these transactions involves counter parties that would smile my way and say "it's on the house".
I'm not suggesting that is likely, but that it's more what would/could happen -- there is already a real, working, full product that wouldn't disappear over night. And a commercial fork would likely be pretty painful for everyone, just as closed Solaris is looking less and less interesting as both many of the minds behind the great parts of Solaris work on Open Solaris in one form or another, and as fewer community resources go into closed Solaris.
I see the AGPL (as opposed to some ad-hoc license) as another benefit for RethinkDB. Many might not like the copyleft-part -- but at least it is a known and well-documented quantity -- no surprises likely to come from/with forking and/or when trying to merge with other Free software (be that BSD or GPL or...).
(BTW the Open Source Institute was unable to get a trademark for "open source" so it doesn't matter that they approved it.)
Now, one can be in agreement with the idea that the four freedoms are important, especially as we increasingly live in a world where software is not only convenient, but necessary in our daily lives -- but the idea with the AGPL, and why it is needed for server software -- is pretty clear.
Firstly, as Daniel pointed out, RethinkDB is licensed under AGPL. An acquirer wouldn't have the legal means to close the source code, and with over 700 forks on GitHub they also couldn't do it practically.
But beyond licensing, consider our personal motivations. We've been working on RethinkDB for five years, and had quite a few opportunities to sell the company. We turned them all down because we really believe in the product. The world is clearly moving towards realtime apps, and we feel it's extremely important for open realtime infrastructure to exist. It's easy for people to make promises about the future, but consider this from a game-theoretic point of view. If we wanted to sell, we could have done it long ago. I know it's not a guarantee, but hopefully it's a strong signal to help with your decision.
(Also, there are lots of really interesting companies building products on RethinkDB that we can't talk publicly about yet. It would be silly to sell given that momentum)
At some point people need bread and butter, so I'm curious where that's going to come from :)
<3 Rethink.
They had only open-sourced the SQL-Layer on top of their key/value store and it's still available on Github. The reason: They build it based on open-source code
When someone deletes a public repository on Github, one fork remains as the new master. (Here's FoundationDB's SQL-Layer: https://github.com/louisrli/sql-layer)
So: RethinkDB will stay, even if someone tries to pull the plug. Just fork them on Github. :)
I've started to look into RethinkDB in the past, and I'm very interested in the features it claims. However, I only have so much time to investigate new primary storage solutions, and our team has been burned in the past by jumping too quickly on a DB's bandwagon when the reliability, performance, or tooling just wasn't there.
As of late, we've come to rely on Aphyr's wonderful Call Me Maybe series[0] as a guide for which of a DB's claims are to be trusted and which aren't. But even when Aphyr hasn't tested a particular DB himself, some projects choose to use his tool Jepsen to verify their own claims. According to at least 1 RethinkDB issue on Github, RethinkDB still hasn't done that[1].
Not to poo poo on the hard work of the RethinkDB team, but for me, the TL;DR is NJ;DU (No Jepsen, Didn't Use)
See the issue you mentioned https://github.com/rethinkdb/rethinkdb/issues/1493 for progress on this.
This is a great point, and we're on it! We have a Raft implementation that unfortunately didn't make it into 2.0 (these things require an enormous amount of patient testing). The implementation is designed explicitly to support robust automatic failover, no interruptions during resharding, and all the edge cases exposed in the Jepsen tests (and many issues that aren't).
This should be out in a few months as we finish testing and polish, and will include the results of the Jepsen tests. (It's kind of unfortunate this didn't make it into 2.0, but distributed systems demand conservative treatment).
I also really like the ability to do joins, where before in Mongo I would have to handle data joins in the app level.
I'm kind of enamoured with the idea of couchapps -- but I'm still not entirely comfortable with having my db be my web and app server, as well as having it manage passwords etc... as I'm reading up, I'm slowly convincing myself it's possible to both make it work, be easy, support a sane level of TLS, load balance and be secure with proper ACL support... but very few tutorials/books seem to really deal with that to a level that brings me confidence.
RethinkDB's realtime capabilities would fit perfectly with Meteor.
You can find out more about the LiveUpdate core project of Meteor on their site [2] - it basically says the implementation of Live Updates for each db driver is independent to what the db is capable of. Specific mention of RethinkDB and Firebase is made as DBs that are built with making realtime data something that you get for relatively little work.
[1] https://github.com/meteor/meteor/blob/devel/packages/mongo/o...
Do you expect that as you stabilize you'll officially support more drivers? Or are you going to leave that as a community effort?
We're planning to take the most well-supported community drivers under the RethinkDB umbrella (assuming the authors agree, of course). It will almost certainly be a collaboration with the community, but we'll be contributing much more to the community drivers, supporting the authors, and offering commercial support for these drivers to our customers.
I've also used Revise a lot and it was great, but the author is busy and hasn't been able to keep up with RethinkDB releases so its now a few versions behind.
Rough numbers you can expect for 1KB size documents, 25M document database: 40K reads/sec/server, 5K writes/sec/server, roughly linear scalability across nodes.
We should be able to get the report out in a couple of days.
The last time I tried with 1.16, I gave up my testing when even the simplest aggregation query (count + group by with what should be a sequential, streaming scan) took literally minutes with RethinkDB, compared to <1s with PostgreSQL. Rethink coredumped before I gave it enough RAM, after which it blew up to around 7GB, whereas Postgres uses virtually no RAM, mostly OS buffers.
Would you mind writing me an email with your query or opening an issue at https://github.com/rethinkdb/rethinkdb/issues (unless you have already?)? I'd like to look into it to see how we can best improve this.
We're planning to implemented a faster count algorithm that might help with this (https://github.com/rethinkdb/rethinkdb/issues/3949), but it's not completely trivial and will take us slightly longer to implement.
select path, count(*) from posts group by path;
You can demonstrate RethinkDB's performance issue with any largeish dataset by trying to group on a single field.
The path column in this case has a cardinality of 94, and the whole dataset is about 1 million documents. Some rows are big, some not; each has metadata plus a JSON document. The Postgres table is around 3.1GB (1GB for the main table + a 2.1GB TOAST table). Postgres does a seqscan + hash aggregate in about 1500ms.
It's been months since I did this, and I've since deleted RethinkDB and my test dataset.
Postgres knows about the structure of your data and where it's located, and can do something reasonably optimal. A generic map/reduce algorithm will have to calculate the same thing as Postgres eventually, but it'll have tons of overhead.
(Also, what is with the fad for running map/reduce in the core of the database? Why would this be a good idea? It was a terrible, performance-killing idea on both Mongo and Riak. Is RethinkDB just participating in this fad to be buzzword-compliant?)
The reason you run queries in "the core" of a database is because copying all the data outside the database and doing computations there would be far worse.
select foo, count(*) from bar group by foo
Mapreduce works great if you are already sequentially churning through a large subset of a table, which is typically the case with aggregations such as "count" and "sum". Where mapreduce is foolish is when you try using mapreduce for real-time queries that only seek to extract a tiny subset of the dataset.
Postgres will still have better numbers, I'm sure. It has a schema for starters.
Postgres has a schema, of course, but it still needs to look up the column map (the ItemIdData) in each page as it scans it, the main difference being that this map is of fixed length, whereas in a schemaless page it would be variable-length.
Anyway, I'm hoping RethinkDB will get better at this. I sure like a lot about it.
table.groupBy(function (x) { ... }).count()
Do you remember if you had set an explicit cache size back when you were testing RethinkDB?
Thank you for the fantastic product, by the way! :)
I realize these numbers alone are still not very meaningful and there are many remaining questions (size and structure of the data set, exact queries performed etc). Rest assured that all of these details will be mentioned in the actual performance report that should be up soon.
One way to improve writes is to batch them, an example is here.
https://github.com/dancannon/gorethink/blob/master/benchmark...
I believe rethinkdb docs state that 200 is the optimum batch size.
Another way is to enable the soft durability mode.
http://rethinkdb.com/api/javascript/insert/
"In soft durability mode RethinkDB will acknowledge the write immediately after receiving and caching it, but before the write has been committed to disk."
https://github.com/dancannon/gorethink/blob/master/benchmark...
Obviously your business requirements come into play. I prefer the Hard writes because my data is important to me but I do insert debug messages using soft writes in one application I have.
*Edit: Heh I forgot to mention, on my Macbook Pro I was getting 20k w/s while batching and using soft writes.
Individual writes for me are hovering around 10k w/s on the 8 cpu 24gb instance i have. But yeah, define your business reqs then write your own benchmarks and see if the need is met.
Many devs write benchmarks in order to be the fastest and not the correctest. Super lame.
We're currently using Mongoid (MongoDB ORM), and an Active Record like ORM for RethinkDB is the main thing holdings us back.
I don't have great insight into nobrainer, but last I checked it seemed like joins wheren't implemented (but on the roadmap).
Nobrainer orm wasn't fun though, too many edge cases that interfere with activerecord and rails conventions. Going a bit on a tangent, after many experiments I've developed a strong conviction that pg is the best database choice for rails, especially with the jsonb datatype included in 9.4. It is the best of two worlds: reliable, proven sql db that plays really well with Rails and has nosql capabilities, including indexing and quering. So good. Ymmv.
That can be a good thing or a bad thing.
People say this a lot, but in our case we really haven't seen this incentive for a couple of reasons.
Large organizations are more than happy to pay for training and development support to accelerate their time to market. It doesn't matter how polished your product is -- databases are complex enough that people are willing to pay for best practices, training, and support.
Similarly, databases are pretty critical pieces of the infrastructure. If anything goes wrong, it can significantly impact the business, so people always want operational/production support.
There are many enterprise services that can be built on top of the product that can be very valuable. You don't have to build a crappy product -- there are plenty of ways to monetize with a great product.
Finally, a bad product will significantly limit growth of the company in the long term. There are lots of options now -- you can't get away with building a crappy product and an artificial monopoly.
If you see a crappy product from a company that offers subscription support, it's probably not because of misaligned incentives. Building databases is really hard, I don't think the business model has much to do with it.
Have I missed something?
As I understand it mongodb have changed the default settings (probably why someone downvoted you) -- but the fact that it was off by default is still something that is rightfully hard for the team to live down.
And while a lot of people are probably still happily using MySQL -- I personally see little use for it, when PostgreSQL is an option.
I maybe wrong, but I think both mongodb and mysql appeal to the same groups: people that don't know or care about normalization, databases and datastructures -- and really just want image (as in Smalltalk) based development, but has been tricked into using php/javascript etc.
It's kind of crazy that you have two mature (one Free, one free) object databases that have seen some real-world usage -- and neither get any love.
One is zodb, the Z object database, developed for zope/plone -- one of the first web application frameworks -- and a major contributor to python (invented eggs, buildout...). It's ridiculously easy to use outside of zope/plone/pyramid[1] and now has a free replication service[2].
The other one is gemstone glass[3] which works with Smalltalk and have their own ruby runtime, maglev[4].
[1] http://zodborg.readthedocs.org/en/latest/documentation/artic...
[2] http://www.zope.com/products/x1752814276/Zope-Replication-Se...
BZ rethinkdb team.
For more information check out https://github.com/dancannon/gorethink/releases/tag/v0.7.0.
Similarly if I have to rebuild a node from scratch, is there a way to prime it so that a massive copy of all the data in the cluster gets copied to it from the other nodes?
Just the delta. We built an efficient, distributed BTree diff algorithm. When a node goes offline and comes back up, the cluster only sends a diff that the node missed.
> Similarly if I have to rebuild a node from scratch, is there a way to prime it so that a massive copy of all the data in the cluster gets copied to it from the other nodes?
You don't have to do that, it happens automatically. You can have full visibility and control into what's happening in the cluster -- check out http://rethinkdb.com/docs/system-tables/ for details on how this works.
Well, in a past life, I used another store that did that automatically, the issue with that is that EITHER it kills the cluster because of read-congestion as it re-builds the "new" node, OR, if you limit the bandwidth for node-building, it takes for ever and a half to rebuild a node which means that you are exposed with one less shard of what was on that node.
What are the chances of a filesystem snapshot to be consistent enough to be used to prime a crashed node? What about restoring backup files from other nodes?
There is currently no other way to prime the node -- I hope we don't have to add it. This sort of functionality should work out of the box.
If you have any further questions I would be more than happy to answer them on https://gitter.im/dancannon/gorethink. Thanks!
Congrats to the RethinkDB team!
https://github.com/fanout/leaderboard/blob/master/tunnel.py
In particular, it reads the entire SSH private key as an environment variable, so you don't need to commit the key to the git repository.
Is windows support coming anytime?
There are reasons to write TypeScript definitions for documentation generation too, if not for the code as TS.
Edit: And Guido is pushing it to Python with PEP 484: https://www.python.org/dev/peps/pep-0484/
It's inherent problem with dynamic languages, you have to read all new release documents and migrate your code. With typed code I at least can be somewhat sure I'm not using deprecated calls and such just by compiling.
Even though there are some well-researched algorithms for it, actually implementing transactions in a distributed system is pretty hard. It also comes at significant performance costs, which would interfere with our goal of easy and efficient scalability.