Ed: i suppose various document/db applications - like IMAP might be a good fit too?
Ed: i suppose various document/db applications - like IMAP might be a good fit too?
My understanding is object stores are typically “flat” by design to scale well (in contrast to a tree structure found in filenames).
For content addressing, are people using the keys in sophisticated ways or are the values being indexed? Any reason to push this complexity into the storage layer as opposed to composing the functionality?
Thats exactly what your last questions points to. If you are lets say dropbox and have probably a super sophisticated key value store setup i can imagine you would want your content addressable layer to be as simple and narrowly optimized as possible and develop and optimize the indexing, metadata and key queries system nearly completely separately. If you are working on some system that also should scale down to run on individual machines like ipfs, git annex or minio before their focus on kubernetes you want a system that can run as a single daemon but also where users can reason about the whole system as an integrated concept.
I used ElasticSearch and run one cluster in production in the past and found it horrible. Maybe I'm missing something, since they're so successful even as a public company...
But I do think that a well-funded and skilled startup team could take a run at ES. They’re a monopoly in their niche. There has to be money in disrupting them.
Shard migration and duplication was a beauty to watch when it worked well, but very often it wouldn't. I tried to establish a procedure to bring instances back up safely and repeatably, but sometimes I would just have to try and try again, no procedure guaranteed restoring operation in all cases. I do remember spending more than one weekend babysitting that cluster. There's a special place in my heart for that memory, and it's not a beautiful one.
I was very surprised to see them IPO successfully - this test was before they were public. I assumed the open source version was just missing the kind of tools and know how to be able to work solidly without paying them or their consultants. These days, I think it's poor engineering resulting in brittleness, we shouldn't have put so much traffic to it, and probably everyone out there experiences the same behavior unless they ensure much safer margins, which probably makes it expensive but I guess who's complaining.
I ended up designing and writing my own (much simpler) distributed system for what I needed (much simpler than everything ElasticSearch does, of course), and it's been in reliable operation since summer 2015. Good engineering goes a long way. Now in the process of turning that approach into a startup. If I get 1/10th as successful as ES I will be a happy camper :)
The good thing about Lucerne existing is you’re allowed to also use their good ideas.