I was surprised to see LevelDB (https://code.google.com/p/leveldb/) was missing from the list of storage solutions you tried, because it seems optimal for your use-case. Were you aware of it?
leveldb on the other hand, supports concurrent writes and provides features to handle data consistency and cheap gradual reindexing.
If you're interested in the internals of LevelDB, I strongly recommend watching this talk http://www.youtube.com/watch?v=vo88IdglU_8 (slides here https://speakerdeck.com/basho/optimizing-leveldb-for-perform...)
A quick glance over LevelDB's features gives me the impression that its bulk-write performance would not be sufficient.