[1] https://github.com/rescrv/HyperDex/blob/master/m4/anal_warni...
270 karma · joined February 17, 2012
[1] https://github.com/rescrv/HyperDex/blob/master/m4/anal_warni...
Your random quotes from man pages are out of context (and keep changing as you edit your posts).
Additionally, the statement you quote explicitly relates to private mappings. LevelDB uses shared mappings.
I took a look at them, and figured out what in the write path could have gone wrong to create the bad behavior. It turned out to be in db/log_writer.cc and deps.
I cannot promise there are no other bugs within Bitcoin or LevelDB, but this resolves one issue. I'm interested in seeing it tested in the wide area, just to make sure.
Of course, if there are other bugs, I'm just as happy to debug and squash them.
autoreconf -I
./configure
makeWhat I can see from their code, is that they rely on a compaction strategy that is very similar to stock LevelDB (in terms of selecting the next uncompacted SSTs within a level), that they haven't done anything to improve multi-threaded performance, and that they've invested a lot of work into computing if/when writes should be delayed within the LevelDB code. We've drastically changed the compaction strategy, begun to improve concurrency (I know for a fact that we have opportunities to improve it further), and we believe that throttling writes at the storage layer is not the correct level to make such decisions, so we've removed the code which does so.
I guess the most fair thing to say is that we've taken complementary approaches, and nothing from either approach is not portable to the other.
1. Our internal synchronization mechanism is a simple change. The stock LevelDB does the following:
place our current on the back of wait_queue
wait for (our thread to be head of wait_queue || thread ahead of us to do our work)
if work done: exit
possibly build a batch of our writes and
append data to the log
insert data into the memtable
signal the next writer, and any writer whose work we finished
HyperLevelDB does this a little differently. We made the log and the memtable concurrent datastructures, so that multiple threads can write to each one at a time. We then do a little synchronization to ensure that we don't reveal the writes to readers in the wrong order. get a ticket, indicating the order of our writes
insert the data into the log
insert the data into the memtable
wait for writes with a lower token to complete
For the actual implementations, check out the code for LevelDB (Lines 1135-1196 of https://github.com/rescrv/HyperLevelDB/blob/28dad918f2ffb80f...) and HyperLevelDB (Line 1307-1428 of https://github.com/rescrv/HyperLevelDB/blob/master/db/db_imp...).Effectively, this change moves from a model where there is exactly one writer at a time, to one where the bulk of the work (inserting into log/memtable) is done in parallel by writer threads.
2. LevelDB provides a GetProperty call. We can inspect the number of files in Level-0 and back-off where appropriate. There is no write delay in LevelDB itself. By the end-to-end principle, the storage server is in a better position to decide whether to delay writes, or just keep pushing them into the database.
The main issue they had was YCSB's method of storing multi-attribute objects as maps (from attribute names to their values) and the fact that Redis cannot support search across keys so YCSB searches within a single set.
In the grander scheme of things, the two systems are very different. In my opinion, Redis is very well suited for inter-process communication where extremely low latency is necessary and data is ephemeral. HyperDex, on the other hand, is great at storing persistent data. The configurable fault tolerance setting makes it very clear how safe your data is, and the strong consistency is very convenient when building applications. Further, HyperDex is built from the ground up to operate in a clustered environment and does so quite well.
2. Currently you cannot perform OR logic in searches. It'd be relatively easy to add though, so we'll probably do so in the future. It's possible to do two searches concurrently and combine the results.
3. This is an oversight on our part. We have to document it. In the Python bindings, you call it the same way as a regular search, but instead of getting a generator that yields objects, you simply get a number.
4. Currently, things are pretty static. We've got ideas for changing that, but just haven't pursued it yet. Each subspace replicates the data again. The memory cost of a subspace is nearly nothing (just a list of replicas).
5. I think that one-level group-by might be achievable, similar to our sorted_search primitive. There's a lot of research that people have done on this topic, and it's very interesting. That being said, the simple/straightforward approach would probably be what we'd take.
Thanks for following the project!
- A SEARCH is simply a mapping from secondary attributes to objects and it was designed for your use case.
- We're still adding server-side atomic operations. In the future we'll definitely have richer types and operations.
- Server-side exec of Lua code is definitely an interesting feature.
- We've put a lot of effort into making it easy to manage a HyperDex cluster without sacrificing on the scalability or fault tolerance. When you do your assessment comparing the two systems, please let us know the outcome because we'll definitely correct any deficiencies.
- The client does indeed maintain a mapping for hashing. Our design is pretty asynchronous already (from the C API), so it'd be a good idea to optimize that further. Of course, we'll eventually document the wire protocol for those who want to go further.
Feel free to contact me if you have questions or need a hand.