TokuMX 1.4: Major improvements to MongoDB sharding and replication

From where I'm standing it appears that 10gen is still iterating fast on features in MongoDB. They're trying to figure out text search, they're expanding the aggregation framework rapidly (which looks really interesting), they're adding security features like auditing, etc. etc.

They have some unfortunate behaviors that are all attributable to their storage implementation, most prominently locking, fragmentation, and slow performance out of memory. While they could "just buy TokuMX" and solve these problems with money, it would then put their engineering team in a position where they would need to relearn a big portion of their codebase, and spend time backporting features they've prototyped to TokuMX. It would basically halt new development for a few months while they learn the new code, too.

The way I see it, MongoDB will continue prototyping interesting features and polishing some of their existing ones, and TokuMX will incorporate the ones with the most promise. But to integrate the codebases would slow down MongoDB considerably, and I don't think they can afford that right now. I'm perfectly happy to sit back and merge the best features from MongoDB as they mature.

Put another way, if you were working on a product and someone came to you and said "here let me fix a bunch of things by replacing some of the fundamental subsystems with code you don't know," would you do it? Maybe if you were in more of a maintenance mode, you'd evaluate it for a while and take the time to learn the code and eventually incorporate it, but not if it was going to distract you from adding features.

Based on my conversations with their CTO, 10gen (nee MongoDB Inc.) is philosophically against what Toku is up to in terms of indexing and optimization.

They're trying to generalize while Toku aims at very specific query optimization.

This isn't a great answer, but I think I owe you an honest one. Our marketing department wants to be able to throw the "please put your email here if you want" form up before a download. We are trying to find a way to reconcile our sales/marketing goals with what we know are our fellow engineers' needs and we hope to make downloads easier in the future. Providing packages at all, over just a single binary tarball, is a step in the right direction, I think.

For now the best I can do for you is tell you that if you email me I can hook you up. Short of that, if you search twitter for "severalnines wget" you can find a wget hack that achieves the result you want.

It's unclear exactly what you mean by "rethinkdb-style" because that could mean a number of things, but stay tuned for our posts on this next week, I think you'll be pleasantly surprised.

ddorian, Can you elaborate what that means?