HNHacker News
TopNewBestAskShowJobs

rescrv

270 karma · joined February 17, 2012

https://rescrv.net
submissionscomments
rescrv··on What Are Your GCC Flags?
We use quite a few flags to issue warnings for our code[1]. The linked autoconf macro will detect whether or not the flags are supported on the detected compiler, and automatically add the flags to the build. It works for clang, gcc, g++, and helps us spot errors that would otherwise be hard to detect and fix.

[1] https://github.com/rescrv/HyperDex/blob/master/m4/anal_warni...

rescrv··on Replicant: Replicated State Machines Made Easy
I'm one of the authors of this post and am happy to answer any questions people may have about Replicant.
rescrv··on Claiming Bitcoin's Bug Bounty
It uses shared writable mappings to reflect writes in the file on disk, and the core purpose of msync is to synchronize the file with the memory map.

Your random quotes from man pages are out of context (and keep changing as you edit your posts).

rescrv··on Claiming Bitcoin's Bug Bounty
Can you link to the man page you are reading, because this is not in the Debian page for mmap(2), which includes munmap.

Additionally, the statement you quote explicitly relates to private mappings. LevelDB uses shared mappings.

rescrv··on Claiming Bitcoin's Bug Bounty
Gavin posted a link to some corrupt LevelDB instances here: https://github.com/bitcoin/bitcoin/issues/2770 He also sent the same link to me via email.

I took a look at them, and figured out what in the write path could have gone wrong to create the bad behavior. It turned out to be in db/log_writer.cc and deps.

I cannot promise there are no other bugs within Bitcoin or LevelDB, but this resolves one issue. I'm interested in seeing it tested in the wide area, just to make sure.

Of course, if there are other bugs, I'm just as happy to debug and squash them.

rescrv··on LevelDB for Ruby
There's also the HyperLevelDB fork which outperforms the Basho one by quite a bit for random write workloads. Here's a write up about it: http://hackingdistributed.com/2013/06/17/hyperleveldb/
rescrv··on Inside HyperLevelDB
We've kept the API is compatible with LevelDB. You just need to change the imports and link against libhyperleveldb.
rescrv··on HyperLevelDB: A High-Performance LevelDB Fork
Please look at the graphs again. Each comparison is done twice: once on hdd and once on ssd. This gives an apples-to-apples comparison for each platform.
rescrv··on HyperLevelDB: A High-Performance LevelDB Fork
We will try to push upstream those pieces that upstream would need. Some pieces, such as the autoconfiscated build system are likely not of interest to them, but of interest to us.
rescrv··on HyperLevelDB: A High-Performance LevelDB Fork
Here's a more-robust way to bootstrap:

    autoreconf -I
    ./configure
    make
rescrv··on HyperLevelDB: A High-Performance LevelDB Fork
I've not fully compared the code, but at first glance, they're choosing different constants for compaction (such as level size, and ratio of compacted-to-uncompacted data) and have done-away with the non-overlapping invariant for the first few levels of the tree. I've not benchmarked their code, so I don't know how effective this strategy is.

What I can see from their code, is that they rely on a compaction strategy that is very similar to stock LevelDB (in terms of selecting the next uncompacted SSTs within a level), that they haven't done anything to improve multi-threaded performance, and that they've invested a lot of work into computing if/when writes should be delayed within the LevelDB code. We've drastically changed the compaction strategy, begun to improve concurrency (I know for a fact that we have opportunities to improve it further), and we believe that throttling writes at the storage layer is not the correct level to make such decisions, so we've removed the code which does so.

I guess the most fair thing to say is that we've taken complementary approaches, and nothing from either approach is not portable to the other.

rescrv··on HyperLevelDB: A High-Performance LevelDB Fork
I'm the HyperDex developer who did the work on HyperLevelDB. I'll answer your questions in order as best I can.

1. Our internal synchronization mechanism is a simple change. The stock LevelDB does the following:

    place our current on the back of wait_queue
    wait for (our thread to be head of wait_queue || thread ahead of us to do our work)
    if work done: exit
    possibly build a batch of our writes and 
    append data to the log
    insert data into the memtable
    signal the next writer, and any writer whose work we finished
HyperLevelDB does this a little differently. We made the log and the memtable concurrent datastructures, so that multiple threads can write to each one at a time. We then do a little synchronization to ensure that we don't reveal the writes to readers in the wrong order.

    get a ticket, indicating the order of our writes
    insert the data into the log
    insert the data into the memtable
    wait for writes with a lower token to complete
For the actual implementations, check out the code for LevelDB (Lines 1135-1196 of https://github.com/rescrv/HyperLevelDB/blob/28dad918f2ffb80f...) and HyperLevelDB (Line 1307-1428 of https://github.com/rescrv/HyperLevelDB/blob/master/db/db_imp...).

Effectively, this change moves from a model where there is exactly one writer at a time, to one where the bulk of the work (inserting into log/memtable) is done in parallel by writer threads.

2. LevelDB provides a GetProperty call. We can inspect the number of files in Level-0 and back-off where appropriate. There is no write delay in LevelDB itself. By the end-to-end principle, the storage server is in a better position to decide whether to delay writes, or just keep pushing them into the database.

rescrv··on HyperLevelDB: A High-Performance LevelDB Fork
We're doing our best to keep API compatibility with LevelDB and will do our best to keep true to LevelDB. That being said, it'd be great to see our changes flow upstream.
rescrv··on HyperLevelDB: A High-Performance LevelDB Fork
So we scale up to (num_cores - 1). We have 8-way and a 4-way machines. I'll update the page to make this clear.
rescrv··on RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements
And each of those individual operations must be preserved. The example of account credit/debit is not used because it reflects reality 100%, but because its a simple-to-understand example to illustrate the power of transactions. Most developers should be able to look beyond that simplification and see how they can put transactions to work (or see that they are unnecessary).
rescrv··on HyperDex brings ACID transactions to NoSQL
I am the lead developer on HyperDex Warp so hopefully I can answer any questions for you. We've put many sleepless nights into testing the code base and are always looking to improve our feature set and reliability.
rescrv··on Cassandra, MongoDB, and HyperDex Performance
There's ongoing work to write one. Tibor Vass started hacking on it [1] awhile back. Scott Dunlop (swdunlop?) has recently started working on finishing the client. Hopefully he'll notice this and reply so maybe you won't have to write your own...

1. https://github.com/tiborvass/hyperclient

rescrv··on Cassandra, MongoDB, and HyperDex Performance
It's also worth noting that Cassandra and MongoDB offer strictly weaker fault tolerance than HyperDex in these benchmarks.
rescrv··on Cassandra Performance
When benchmarking HyperDex we found something similar. Cassandra is better at writes than reads by a surprising amount. Given Facebook's elaborate caching architecture (which likely pushed for such an inversion) this makes sense.
rescrv··on Cassandra Performance
Checkout HyperDex. We just released a new version, and it is indeed consistent as nodes join and fail.
rescrv··on Cassandra Performance
Consistency is a safety property. HyperDex, Hbase and BigTable all provide linearizability, which has a well-defined meaning. Cassandra does not, and most of its descriptions of consistency only refer to the behavior of the system, and not the properties you can rely upon. Pointing to the number of replicas read or written only clouds the issue.
rescrv··on Cassandra Performance
Check out HyperDex, Hbase and BigTable for systems that provide better guarantees than "eventually."
rescrv··on HyperDex, a consistent, fault-tolerant key-value store, hits version 1.0.rc1
I bet you found the thread on their mailing list. I was the one who ran the benchmark and it was one core vs. one core. In the other benchmarks we had multiple machines, but not the Redis benchmark. We clarified after they speculated that a multi/single comparison was performed.

The main issue they had was YCSB's method of storing multi-attribute objects as maps (from attribute names to their values) and the fact that Redis cannot support search across keys so YCSB searches within a single set.

rescrv··on HyperDex, a consistent, fault-tolerant key-value store, hits version 1.0.rc1
First, HyperDex is well-optimized C++ code. And second, they take issue with the standard benchmark we used to evaluate both systems. While we recognize that Redis may have an impedance mismatch with the benchmark, we were hesitant to create our own lest we inadvertently introduce bias toward our system. Thus the numbers reported are accurate, but the generic nature of the YCSB benchmark may not be exploiting all that Redis offers. We'd love to work with the Redis team on a standardized benchmark we both can get behind but that has not yet happened (and may never happen).

In the grander scheme of things, the two systems are very different. In my opinion, Redis is very well suited for inter-process communication where extremely low latency is necessary and data is ephemeral. HyperDex, on the other hand, is great at storing persistent data. The configurable fault tolerance setting makes it very clear how safe your data is, and the strong consistency is very convenient when building applications. Further, HyperDex is built from the ground up to operate in a clustered environment and does so quite well.

rescrv··on HyperDex, a consistent, fault-tolerant key-value store, hits version 1.0.rc1
1. In the current release it's possible to completely shutdown the cluster. You cannot just cut power, but it's relatively painless to do otherwise. Checkout http://hyperdex.org/doc/06.faults/#shutting-down-and-restori... for more details.

2. Currently you cannot perform OR logic in searches. It'd be relatively easy to add though, so we'll probably do so in the future. It's possible to do two searches concurrently and combine the results.

3. This is an oversight on our part. We have to document it. In the Python bindings, you call it the same way as a regular search, but instead of getting a generator that yields objects, you simply get a number.

4. Currently, things are pretty static. We've got ideas for changing that, but just haven't pursued it yet. Each subspace replicates the data again. The memory cost of a subspace is nearly nothing (just a list of replicas).

5. I think that one-level group-by might be achievable, similar to our sorted_search primitive. There's a lot of research that people have done on this topic, and it's very interesting. That being said, the simple/straightforward approach would probably be what we'd take.

Thanks for following the project!

rescrv··on HyperDex, a consistent, fault-tolerant key-value store, hits version 1.0.rc1
You don't need all machines in the same data center. Latency between machines does have effects on performance, so it is faster in a single data center.
rescrv··on HyperDex, a consistent, fault-tolerant key-value store, hits version 1.0.rc1
We love Python and it makes for easy-to-read code snippets! Internally we do make use of the fact that LevelDB sorts objects by key.
rescrv··on HyperDex, a consistent, fault-tolerant key-value store, hits version 1.0.rc1
Thanks for your time. I'm the lead dev on HyperDex, so I'd be happy to address your points.

- A SEARCH is simply a mapping from secondary attributes to objects and it was designed for your use case.

- We're still adding server-side atomic operations. In the future we'll definitely have richer types and operations.

- Server-side exec of Lua code is definitely an interesting feature.

- We've put a lot of effort into making it easy to manage a HyperDex cluster without sacrificing on the scalability or fault tolerance. When you do your assessment comparing the two systems, please let us know the outcome because we'll definitely correct any deficiencies.

- The client does indeed maintain a mapping for hashing. Our design is pretty asynchronous already (from the C API), so it'd be a good idea to optimize that further. Of course, we'll eventually document the wire protocol for those who want to go further.

Feel free to contact me if you have questions or need a hand.

rescrv··on HyperDex, a consistent, fault-tolerant key-value store, hits version 1.0.rc1
EC2 is just one platform. Although many people point to EC2 as the gold standard, it's worth noting that Google and Facebook both avoid virtualized servers. On the other hand, EC2 is about leveraging spare capacity and intentionally oversubscribing machines.
rescrv··on Inside Google Spanner, the Largest Single Database on Earth
TrueTime measures a point in time plus/minus an error interval. The API allows a program to query for when a time/error pair has definitely passed. Each transaction acquires locks using 2-phase locking. At commit time, the time/error are read and the decision to commit is stored durably using Paxos. The locks are then held until the time of the transaction has definitely passed. This guarantees that no process which grabs the locks after they are released can commit at an earlier time than previous holders of the lock.
← PreviousPage 2 of 4Next →