STM, CouchDB, and writing 5500 documents/second

It's quite durable. CouchDB has pure tail append storage and will fsync every update and batch concurrent updates into a single commit. It can get fairly close to a drives max sustained write speed.

Except that's always the case. You mean if Clojure crashes before it queues up 50 docs and sends them, the docs in the queue are lost.

Durability is a concern when it means the integrity of your data is unknown. You'd probably be doing something like this in a bulk load, in which case you'd know where it crashed.

If this were queuing and batching inserts from many users it might be more of an issue (a user thinks something happened and it didn't), but realize that the at a queue size of 50 this performs at about ~5500 inserts/second. (And you can reduce this queue size if you don't need to insert that fast) This would affect users who happen to make their request within a roughly 0.18 millisecond window. If confirmation is still absolutely important, the post also links a version at the end that -does- wait for and return the IDs of the inserted documents from couchdb, guaranteeing the inserts have happened, and it still performs quite well.

But not much more data than a webserver crash in a fully concurrent situation. One enhancement that I didn't bother to implement because it would muddy up the example was a timed flush so that data never just sits around in memory because the queue isn't full.

For me, the problem has always been the indexed views taking too long to generate for a large amount of documents on first view. I know that some people do a bulk update, then hit the different indexed views to get them updated incrementally.

What can be done to increase the speed of the first view of new indexed views that you might have later on? Do you just have to wait for it?

What about ad-hoc views? Are there ways to speed that up?

i find this whole thing a lot more of an experiment in clojure performance than CouchDB throughput.

he doesn't attach a handler to the response on his writes and is queuing writes in memory and bulking them. he's not concerned with durability at pretty much any layer.

this kind of setup is great for a large initial import of data but i would be weary of running it in production.

this was part of my initial reaction.

when CouchDB is under concurrent load it does a "group commit" style write anyway and can batch nearly as efficiently as bulk docs. the only additional overhead would be the HTTP headers which is nominal when the documents are a bit larger than "hello world".

in general CouchDB does better under concurrent write load than single writer and while these kinds of optimizations on the client side can be impressive it's not actually what CouchDB is tuned for OOTB.

the benefit of a bulk index is the ability to shove everything into a file and fsync once. so splitting across files won't really give the performance benefits.

riak does support automatic sharding and enables you to tune your CAP paramters. Getting nodes in and out of a cluster is way easier with riak. While riak does also support map/reduce, generating something like a couchdb's "views" will probably end up being a problem since listing all the keys of large buckets (needed for a view of all your data) is a really expensive operation in riak. Also: couchdb supports incremental views, so if the view is fixed, you'll only need to update the changed documents. My main problem with couchDB was that in systems with a lot of changing data, the compaction was too slow for me. I actually changed data faster than the compaction could work on. Also: you have to compact that stuff yourself by sending an explicit http request.