Jepsen: VoltDB 6.3

First, Raft wasn't a thing when we started.

Second, the consensus algorithm we use today is used for cluster membership, schema changes and other agreement purposes, but not for the main transactional path. However, in versions before 3.0 it was.

At the time, we wanted to make really different tradeoffs than something like Paxos. Namely, we wanted to be able to run with two copies, rather than the three that Paxos/Raft require. We also wanted crazy throughput and low performance variation, which are hard things to get with Paxos/Raft. The main tradeoff is that our system doesn't work well across high-latency links, which is ok with us.

Ultimately we added a second layer protocol for transactions in 3.0, and this made the transaction path not dependent on clock skew. This layer actually looks a lot like Raft, but does differ in some key ways.

We may even someday switch to Raft for the core agreement stuff, but there's a lot of work to be done for that to happen.

This link has more info about how transactions work in VoltDB: https://voltdb.com/sites/default/files/tn-transactions.pdf

Personally, when a node fails, I still want to try to understand why before I take the next operational step.

Thanks for your words, BTW.

But my more typical answer is the combination of throughput and transactional logic. No other system does both as well as VoltDB. This comes up a lot in policy enforcement, fraud detection, online gaming, ad-tech, billing-support and more.

Here are some blog posts that might help:

https://voltdb.com/blog/winning-now-and-future-where-voltdb-... https://voltdb.com/blog/apps-need-acid https://voltdb.com/blog/call-center-example-integrating-proc...

This video's not bad:

https://voltdb.com/resources/video/transactional-streaming-i...

http://www.se-radio.net/2013/12/episode-199-michael-stonebra...

In practice, this isn't always all that useful without a real stored procedure language.

Supporting other ways to make procedures is something we've always planned and at some point we will get to it. I personally like javascript and lua.

For 100% read workloads, the impact to maximum throughput can be significant. That's a pretty uncommon workload for us though. It's likely a single-digit percentage problem at 50% reads.

Two nice things about VoltDB users: 1) They often are very write heavy, 99% read-write transactions isn't uncommon. 2) Few are using anywhere near the maximum throughput for their machines. Most size their clusters for appropriate storage and redundancy, not throughput.

The lost write issues weren't optimizations, just implementation bugs. Sigh.

I'm not sure what you mean by "within the start/end time of the partition executing the transaction". Nothing in the informal definition I provided mentioned partitions, or even process-local orders (that'd be sequential consistency) so ... yeah, I dunno where you got this from. I agree that linearizability is a global real-time constraint, and I use that sense in the post and in the Knossos verifier.

Most distributed databases (from what I can tell, including voltdb) only guarantee that for operations which read/write the same keys.

You raise an interesting question: if I verify only that operations on a single key are linearizable, have I also verified that operations on systems of independent keys are linearizable? The answer, as far as I know, is yes: linearizability is a local (or "composable") property. From Herlihy & Wing (https://cs.brown.edu/~mph/HerlihyW90/p463-herlihy.pdf):

Unlike alternative correctness conditions such as sequential consistency [31] or serializability [40], linearizability is a local property: a system is linearizable if each individual object is linearizable. Locality enhances modularity and concurrency, since objects can be implemented and verified independently, and run-time scheduling can be completely decentralized.

This is a commonly cited property in the literature, and has been proven several ways--for instance, see Lin's recent constructive proof (http://arxiv.org/abs/1412.8324). There is research showing linearizable systems vary in the probability distribution of outcomes (https://arxiv.org/pdf/1103.4690.pdf), but this does not affect safety.

However, your comment led me to Charron-Bost & Cori 2003 (http://www.worldscientific.com/doi/abs/10.1142/S012962640300...), whose abstract claims a counterexample system of two linearizable objects whose composed system is nonlinearizable. I haven't found the full text yet, and I'm not familiar with their sense of "The Global Time Axiom", so it's possible their finding is still consistent with "linearizability is composable". Not sure.

In any case, the multi-key tests in this analysis do perform single-key transactions (as well as multi-key transactions), and verify that their composed system is fully linearizable. Because the state space for composed systems is larger, these tests aren't as good at finding bugs--but if composability turns out not to hold, I can use this strategy more often.

But in the presence of side channels, you need a truly global clock (like spanner) to achieve strict serializablility.

As I understand it, Spanner's global clocks are a performance optimization, not a correctness condition. If linearizability required a global clock, Zookeeper (http://static.cs.brown.edu/courses/cs227/archives/2012/paper...) and Raft (https://raft.github.io/raft.pdf) wouldn't be able to provide linearizable semantics. It is, of course, possible that these papers are wrong, in which case I encourage you to publish!