Once you are dealing with more than 500k keys or so, Riak starts to fall over.
EDIT: The 500k key limit pertains to mapreduce jobs, not the overall data size.
Riak will handle billions of keys just fine. We had, I dunno, a half a billion in a six node bitcask-backed cluster and were only at half capacity. Much much bigger installs exist. The limit I was referring to is for a single mapreduce job; Riak MR just isn't well-suited to operations over millions of keys at a time. It can do it, but Riak MR isn't really designed for bulk processing: and I wouldn't be surprised to see MR become unusably slow over millions of keys. You'll get better performance out of Hadoop, generally, for bulk analytics.
The other tough point is key-listing. Listing buckets, listing keys, key filters, MR over buckets, all those features are essentially useless in production. Where the number of keys is large and unguessable it can become a logistical nightmare to keep track of them. 2I key indexes can help, though.
I want to put millions of items in riak, play with it, and then throw then away.
I might want to do that because I'm testing out something, or because it's the result of some periodic batch processing in production, which I want to get by key later.
Unfortunately, riak doesn't seem to have the notion of a "db", "keyspace" or whatever you want to call it; i.e. something which you could "drop" and that will simply delete a directory with a dozen of files in it (should be quite cheap).
The only thing I can do is to drop the whole riak db, which has the following problems:
1) I have to do it manually on all nodes (stopping the cluster, deleting the files etc)
2) I cannot share a riak cluster between several users/team, so that each user/team can play with a portion of it but there is only a central installtion of the whole cluster. Every application (which I want to be able to drop all the db and recreate it) has to run it's own riak cluster.
Initially I thought that "buckets" were intended to solve this problem, but buckets don't map to a separate storage location, it's just a way to group items. Even listing all buckets present in the db requires scanning all keys and, as the doc says, "Similar to the list keys operation, this requires traversing all keys stored in the cluster and should not be used in production."
Although I've been told that "riak is not designed to do this and that", I'm not sure if these limitations are really technical, or just because the product development effort was targeted at some of the aspects, and these issues could be addressed in a later stage.
Any idea?
a.) Run multiple clusters--not too difficult. Just give each one a different erlang cookie and run em on subsequent ports.
b.) Take bitcask_backend or leveldb_backend and add drop-bucket functionality. Custom backends are more difficult than running multiple clusters, but certainly not impossible. You could build it on top of fold or split writes up into, say, one leveldb per bucket. Don't recall if the vnode interface has drop-bucket so you might have to write some plumbing alongside Riak. jrecursive has done this in Mecha.
If I were building something like this, I might look first at Cassandra or Hbase, or possibly sharded master-slave postgres.
1. Even if it's easy, somebody has to do that. 2. Setting up all the monitoring etc for each instance 3. Running more than one riak daemon on the same machine means that the riak daemon is unaware of the IO operations performed by the other one, hence IO throughput could suffer. This means that in practice you would need to mount separate disk heads (and we are back to 1.) 4. Each riak instance will require some RAM as well, so memory has to be allocated and there is the risk that's over-allocated. 5. Port allocation. I fear it would end up with smth like: "just keep a internal wiki page where each 'db space' is mapped to a port number"
Well, the problem with (b) is of course that I don't have time to do that. For now we stick to cassandra, but Riak is so nice in many aspects that I really hope that at some point, as the product matures, more resources can be invested in aspects which are not currently perceived as "selling points" for riak, but are important for some scenarios and not technically impossible.
You're bashing a product here based on your lack of knowledge, not the products lack of capabilities.
We just saw you over in the programming languages thread, now you're here, refusing to confront the reality of how broken M/R is in Riak.
What monkey crawls around on your back to make you so confrontational and irrational?
Hadoop may have been better for what you are doing, and logging metrics is a particular use case where a specialized database is most appropriate.
But it is incorrect to imply that Riak falls over at any specific key limit. This is simply untrue. With Riak you can always add more nodes if you need more capacity, and Map Reduce is done in a distributed fashion so adding more nodes adds map reduce capacity. Its not perfect but it is not brittle.
http://lists.basho.com/pipermail/riak-users_lists.basho.com/...
"If large-scale mapreduce (more than a few hundred thousand keys) is important, or listing keys is critical, you might consider HBase."
"Riak can also collapse in horrible ways when asked to list huge numbers of keys. Some people say it just gets slow on their large installations. We've actually seen it hang the cluster altogether. Try it and find out!"
You want to say I'm wrong, make a specific argument. Don't selectively quote things out of context that actually don't rebut my position, as that's profoundly dishonest. It is a way of pretending to rebut someone but without saying anything yourself so you can't be pinned on any statements. It is disingenuous.
I'm really tired of having to rebut these argument-from-ignorance "rebuttals" here on HN.
Ctrl-F billions and found one match in the post I was quoting. No other reference to very large MR jobs in the post quoted.
"At Showyou, we're also building a custom backend called Mecha which integrates Riak and SOLR, specifically for this kind of analytics over billions of keys. We haven't packaged it for open-source release yet"
So the OP is supposed to use an unreleased experimental custom backend to do his big mapreduce jobs?