leif

If you're having problems with MongoDB on large data sets, try TokuMX: http://www.tokutek.com/products/tokumx-for-mongodb/

thanks a lot, argv0

For other cmake writers out there, this isn't hard.

    set(LIBDIR lib CACHE STRING "lib dir")
    install(${lib_targets} DESTINATION ${LIBDIR})

That's what I do and I am able to build packages with it just fine, that send the libs to wherever the hell they want them.

CTO is a professor at UNH, Chief Scientist is from UNH and used to work at Virtual Iron which was acquired by Oracle, then he worked at Akiban until they were acquired. It appears some of their patents (http://www.google.com/patents/US20130290243, http://www.google.com/patents/US20130254208, http://www.google.com/patents/US20130226931) were assigned to Cloudtree Inc., which appears to have originally been something totally different but with what seems to be the same team: http://www.boston.com/business/technology/innoeco/2011/02/st...

I'm puzzled.

This links to the wrong fractal tree paper for discussing write-read optimized data structures. The proper one is http://supertech.csail.mit.edu/papers/sbtree.pdf

This makes me wonder how much research this team really did. I'm not ready to claim they haven't done it, but it raises the question. I would like to see the actual data structure published somewhere.

It's telling that those jira tickets are both in the top five most voted on open server tickets and are from 2010 and before. (I know you know this roger, but) TokuMX completely resolves both of them.

I agree. We're putting a lot of effort into this at Tokutek with TokuMX: http://www.tokutek.com/2014/02/introducing-tokumx-1-4-major-...

Personally, I think the company formerly known as 10gen is doing a lot of good work, particularly with the aggregation framework, but it's orthogonal to the improvements we've made with the storage subsystem.

This isn't a great answer, but I think I owe you an honest one. Our marketing department wants to be able to throw the "please put your email here if you want" form up before a download. We are trying to find a way to reconcile our sales/marketing goals with what we know are our fellow engineers' needs and we hope to make downloads easier in the future. Providing packages at all, over just a single binary tarball, is a step in the right direction, I think.

For now the best I can do for you is tell you that if you email me I can hook you up. Short of that, if you search twitter for "severalnines wget" you can find a wget hack that achieves the result you want.

From where I'm standing it appears that 10gen is still iterating fast on features in MongoDB. They're trying to figure out text search, they're expanding the aggregation framework rapidly (which looks really interesting), they're adding security features like auditing, etc. etc.

They have some unfortunate behaviors that are all attributable to their storage implementation, most prominently locking, fragmentation, and slow performance out of memory. While they could "just buy TokuMX" and solve these problems with money, it would then put their engineering team in a position where they would need to relearn a big portion of their codebase, and spend time backporting features they've prototyped to TokuMX. It would basically halt new development for a few months while they learn the new code, too.

The way I see it, MongoDB will continue prototyping interesting features and polishing some of their existing ones, and TokuMX will incorporate the ones with the most promise. But to integrate the codebases would slow down MongoDB considerably, and I don't think they can afford that right now. I'm perfectly happy to sit back and merge the best features from MongoDB as they mature.

Put another way, if you were working on a product and someone came to you and said "here let me fix a bunch of things by replacing some of the fundamental subsystems with code you don't know," would you do it? Maybe if you were in more of a maintenance mode, you'd evaluate it for a while and take the time to learn the code and eventually incorporate it, but not if it was going to distract you from adding features.

All TokuMX data and index storage is block compressed. It's not hard, it's on by default.

Unfortunately, this would break compatibility with existing MongoDB applications more than we would probably be willing to do. However, there's no reason RethinkDB couldn't use Fractal Tree indexing instead of B-trees, given some engineering effort.

It's unclear exactly what you mean by "rethinkdb-style" because that could mean a number of things, but stay tuned for our posts on this next week, I think you'll be pleasantly surprised.

Ok. Well, you said you were waiting for it to be ready, and I think it is. We'll be here when you get a week free to tinker.

If you were using upserts I expect you were having problems due to the optimizer retrying all possible plans (including table scan) periodically. This is reflected in https://github.com/Tokutek/mongo/issues/796 and is fixed in 1.4.0. If you'd like to try another evaluation, get in touch with us and we can help you track down whatever problems you see.

Not all mongodb code will optimally use tokumx without any changes. Concurrency is hard and mongodb encourages some patterns that are bad for any concurrent database. For example, count() for an entire collection is not, and could never be, as cheap in a concurrent database like tokumx as it is in mongodb.

When are these comments going to stop? Everyone lazy enough is going to keep suggesting "curl install" and everyone lazy enough is going to keep using it, and everyone that has something worth protecting and cares enough is either not going to use it or is going to audit it anyway, and comments like this aren't going to move anyone between those camps.

If you all think "curl install" is so evil then build something secure to replace it. It's got to be as easy from a "package maintainer" as well as a user's perspective, and you have to figure out some way to validate security. But until then, warning people that they should read code they execute is a waste of bandwidth.

sudo echo foo > bar doesn't work, instead use echo foo | sudo tee bar

beatbox

couldn't be more thrilled by that

Kind of. TCmalloc used to never unmap pages (though it would reuse things the client freed), which made it awful for long-lived processes with dynamic memory usage.

They're functional alcoholics.

HAT (http://www.bailis.org/papers/hat-vldb2014.pdf) looks pretty promising in terms of multi-term transactions. It turns out that you can push the problem off to garbage collection in order to make transaction id generation easy, and garbage collection is easier to be sloppy and heuristic about. The only problem is it isn't clear yet that HATs are as rich as 2PL-based approaches, and that nobody's built an industrial strength implementation yet.

Oh ok, now I see what you're saying, it's still similar to an insert buffer in that case. B-tree behavior is still to blame for this, and if you make your log file bigger it lets you soak up more writes before you need to checkpoint but you'll either have even longer checkpoints eventually, or you'll run out of memory before you have to checkpoint.

We also checkpoint before trimming the log, but our checkpoints are a lot smaller because of write optimization.

I think you're talking about an insert buffer, not a transaction log, and in that case, no matter how big your insert buffer is, it will eventually saturate and you'll end up hitting the performance cliff of the B-tree. You really need better data structures (like fractal trees or LSM trees) to get past it.

I'm not talking about the speed of the array, rather a battery backed controller. With one of those, as long as you're under the sequential write bandwidth of the underlying array, it pretty much doesn't matter how much you fsync the log, so that bottleneck (commit frequency) goes away.

If you're planning to write data faster than disk bandwidth, then you have no hope of being durable and we're talking about problems too different to be worth comparing, and in that case I retract my comment.

I don't understand what distinction you're trying to make between the "array itself" and the "log file has a limit to how many blocks can be appended". Can you clarify what limit you're talking about?

The bids and offers made in the market set the price, just like all currencies. Right now the market is too small to be stable like most currencies (not enough inertia, too many events if large relative significance, not enough arbitrageurs), but the wild swings you see are not due to the free will whims of the exchange managers, they're due to the market's youth and size.

The write bottleneck for traditional databases has never been the write-ahead log, with group commit and a battery-backed RAID controller you'll have a hard time saturating the disk with log writes. The bottleneck has always been random I/O induced by in-place updating indexes based on B-trees. You don't need to be in-memory if you use better data structures. TokuDB and TokuMX are proof of that.

Ok, that's more drastic than I thought. I think I get the point now.

I'm not a web guy, but isn't sass a CSS generator, as in something you run once before deploying anything? What's the advantage of speed for something like this?

That thread is talking about the possible inclusion of our indexing library inside Postgres, and was from before we had open sourced the code. IANAL but I think it would be possible to include the fractal tree library in Postgres now if someone wanted to do it, but none of it has ever been implemented by Postgres.

I promise it hasn't, but I'm curious to see what you're talking about. Link?

Fine-grained locking is hard, but "tar pit" is unfair and honestly a bad attitude. It's crucial for modern applications, and it can be done if you're careful, and it can be done really well.

We (Tokutek) tried for a long time to get by with a big monolithic lock, and a) competing with InnoDB was really hard since they do concurrent writers really really well, and b) when we did decide to break up the lock, it wasn't as hard as we thought it would be and it worked really really well.

Don't get discouraged, break up that lock!