aphyr

Yeah, that's a good way of phrasing it! :-)

> strict serializability doesn't imply idempotency

I think we're probably getting at the same thing, but I do want to clarify a bit. A Strict Serializable history, like a Serializable one, requires equivalence to a total order of transactions. That's clearly not true for etcd+jetcd: no possible order of transactions can allow (e.g.) a transaction to read from its own future. It's totally fine to submit non-idempotent transactions against a Serializable system: systems which actually provide Serializable will execute known-committed transactions exactly once.

Plenty of other databases pass this test; etcd+jetcd does not. This system is simply not Serializable.

This is also good to hear! I'm not sure whether I'd call it a "footgun" per se--that's really an empirical question about how Datomic's users understand its model. I can say that as someone with some database experience and a few weeks of reading the Datomic docs, this issue actually "broke" several of the tests I wrote for Datomic. It was especially tricky because the transactions mostly worked as expected, but would occasionally "lose updates" or cause updates intended for one entity to wind up assigned to another.

Things looked fine in my manual testing, but when I ran the full test suite Elle kept catching what looked like serious Serializability violations. Took me quite a while to figure out I was holding the database wrong!

This is good to hear! Nubank has also argued that in their extensive use of Datomic, this kind of issue doesn't really show up. They suggest custom transaction functions are infrequently written, not often composed, and don't usually perform the kind of precondition validation that would lead to this sort of mistake.

Yeah, I think this is next to Zookeeper as one of the most positive Jepsen reports. :-)

Thank you!

To build on this, Datomic includes a pre-commit conflict check that would prevent this particular example from committing at all: it detects that there are two incompatible assertions for the same entity/attribute pair, and rejects the transaction. We think this conflict check likely prevents many users from actually hitting this issue in production.

The issue we discuss in the report only occurs when the transaction expands to non-conflicting datoms--for instance:

[Stu favorite-number 41]

[Stu hates-all-numbers-and-has-no-favorite true]

These entity/attribute pairs are disjoint, so the conflict checker allows the transaction to commit, producing a record which is in a logically inconsistent state!

On the documentation front--Datomic users could be forgiven for thinking of the elements of transactions as "operations", since Datomic's docs called them both "operations" and "statements". ;-)

Yeah, this basically boils down to "a potential pitfall, but consistent with documentation, and working as designed". Whether this actually matters depends on whether users are writing transaction functions which are intended to preserve some invariant, but would only do so if executed sequentially, rather than concurrently.

Datomic's position (and Datomic, please chime in here!) is that users simply do not write transaction functions like this very often. This is defensible: the docs did explicitly state that transaction functions observe the start-of-transaction state, not one another! On the other hand, there was also language in the docs that suggested transaction functions could be used to preserve invariants: "[txn fns] can atomically analyze and transform database values. You can use them to ensure atomic read-modify-update processing, and integrity constraints...". That language, combined with the fact that basically every other Serializable DB uses sequential intra-transaction semantics, is why I devoted so much attention to this issue in the report.

It's a complex question and I don't have a clear-cut answer! I'd love to hear what the general DB community and Datomic users in particular make of these semantics.

Yeah, of course :-)

You'll find many of those additional variants (e.g. strong session SI) in the linked papers, and they're cited extensively in the Jepsen reports as well. I just haven't had time to write up every single model--tried to stick to the major ones. :-)

Same. Hi, Apple! :D

I'd love to do more work with predicates in general. That's an open research problem I've been noodling on for years. Pretty much any SQL DB would be a good candidate for that work!

I'm gonna be a weirdo and say I actually loved Fauna's FQL. A little Lisp-ish functional language for queries is a great way to interact with document-structured data. SQL is fantastic for sheer breadth, though its specification is a nightmare and actually writing portable SQL is real challenging. One of those places where a stronger spec and conformance tests would have really helped.

It's not a huge market. I usually have a queue of clients, but both deal flow and actual scheduling are wildly variable--sometimes I'll go without income for six months or more. I've considered hiring one or two people, but I couldn't offer the kind of stability people need from an employer. I do subcontract though! Editing, legal, finance, occasional code, that sort of thing.

There's a lot of folks out there who can do basic testing work with Jepsen. I've taught... I dunno, maybe a few hundred people directly in Jepsen workshops. A couple people have worked alongside me, and I'm sure lots more have learned from the docs online. Writing a report is a more involved problem--certainly not intractable, but for me it involves testing, experiment design, lots of reading, doc review, writing, editing, finding reviewers, and of course all the business stuff.

Even an unpaid report like this involves weeks of work: reading docs, designing tests, executing and refining them, filing issues, writing the report, and editing it. Each report costs thousands of dollars in hardware and and editing. I'd love to do an FDB test! It's just that between contracting, volunteer work, and research, I can only do so much.

The point of isolation levels is that the database manages those locks (or other concurrency control mechanisms) for you.

This depends on what you mean by "atomic". Prior to 5.0, MongoDB's defaults were to use a sub-majority write concern. This allowed all kinds of interesting atomicity violations, even on single documents. For instance, you could write a value, some clients might read it, and then your write would be silently lost as if it never happened. Clients could disagree on whether the write happened or not. A single client could observe, then un-observe the write. It gets weird. :-)

The FUSE stuff actually isn't too bad--Jepsen goes to a lot of trouble to make all this stuff automatic. The test harness pulls dependencies, compiles LazyFS, and mounts the filesystem for you. Just pass `--lazyfs` at the CLI. :-)

Basically all of it. MariaDB exhibits every class of anomaly we found in MySQL, all in single-node deployments.

I, uh, do want to point out that the alternative here is not "everything is OK". If you don't abort when, say, two users update the same row concurrently, then you might cause (e.g.) silent data loss for one of them. Or you might end up with a record in an illegal state--say, one with two different fields that should never be in their particular states together. You have to look at your transaction structure, intended application invariants, and measured frequency of concurrency to figure out if using a relaxed isolation level is actually safe or not.

Aw shucks, thanks :-)

That depends! As the article discusses, snapshot allows anomalies--like write skew--which might violate application invariants. Depends on your workload.

Yup! See https://jepsen.io/analyses/mysql-8.0.34#list-append, which also has a link to the code: https://github.com/jepsen-io/mysql/blob/4c239cb5c66a7f1a55fa...

This isn't CONCAT-specific, BTW--we just use CONCAT because it allows us to infer anomalies in linear, rather than exponential time. Same kinds of behaviors manifest with plain old read/write registers.

Necessary depends on your use case! I spend a lot of time waiting on analyses, and the more operations I can test, the more bugs I can find. I probably invest more time in performance optimization than most people.

Regarding your specific example, uh, I don't know how you measured, but that feels... off. Here:

``` package scratch;

public class Sum { public static long sum(long[] longs) { long sum = 0; for (int i = 0; i < longs.length; i++) { sum += longs[i]; } return sum; } } ```

``` (require '[criterium.core :refer [quick-bench bench]]) (def longs (long-array (range 100000000))) (defn sum-clojure [size] (reduce + 0 (range 0 size))) (import 'scratch.Sum) ```

Summing an array of 10^8 longs like you suggested takes ~120 ms on my machine.

``` user=> (quick-bench (Sum/sum longs)) Evaluation count : 6 in 6 samples of 1 calls. Execution time mean : 119.804166 ms Execution time std-deviation : 9.124709 ms Execution time lower quantile : 115.154905 ms ( 2.5%) Execution time upper quantile : 135.597497 ms (97.5%) Overhead used : 20.384654 ns

Found 1 outliers in 6 samples (16.6667 %) low-severe 1 (16.6667 %) Variance from outliers : 15.4197 % Variance is moderately inflated by outliers ```

Your Clojure function, which sums a lazy range, takes ~4.7 seconds--about 40x slower.

``` user=> (quick-bench (sum-clojure 100000000)) Evaluation count : 6 in 6 samples of 1 calls. Execution time mean : 4.739575 sec Execution time std-deviation : 346.895187 ms Execution time lower quantile : 4.557732 sec ( 2.5%) Execution time upper quantile : 5.324739 sec (97.5%) Overhead used : 20.384654 ns

Found 1 outliers in 6 samples (16.6667 %) low-severe 1 (16.6667 %) Variance from outliers : 15.3163 % Variance is moderately inflated by outliers ```

An idiomatic Clojure reduction over the same array of longs, just so we're measuring apples to apples, takes about 12 seconds--about 100x slower.

``` user=> (reduce + longs) 4999999950000000 user=> (quick-bench (reduce + longs)) Evaluation count : 6 in 6 samples of 1 calls. Execution time mean : 12.037821 sec Execution time std-deviation : 433.661044 ms Execution time lower quantile : 11.790637 sec ( 2.5%) Execution time upper quantile : 12.779262 sec (97.5%) Overhead used : 20.384654 ns

Found 1 outliers in 6 samples (16.6667 %) low-severe 1 (16.6667 %) Variance from outliers : 13.8889 % Variance is moderately inflated by outliers ```

Incidentally, this is one of the reasons I wrote `loopr` (https://aphyr.com/posts/360-loopr-a-loop-reduction-macro-for...). One of the iteration tactics it can compile to is iteration over arrays. Nowhere near Java--I think it's probably still boxing a fair bit, and I'd need to disassemble it to see--but it's still 10x faster than the standard reduce here. ~10x slower than Java.

``` user=> (quick-bench (loopr [sum 0] [x ^"[J" longs :via :array] (recur (+ sum x)))) Evaluation count : 6 in 6 samples of 1 calls. Execution time mean : 1.259095 sec Execution time std-deviation : 3.876650 ms Execution time lower quantile : 1.256692 sec ( 2.5%) Execution time upper quantile : 1.265747 sec (97.5%) Overhead used : 20.321072 ns

Found 1 outliers in 6 samples (16.6667 %) low-severe 1 (16.6667 %) Variance from outliers : 13.8889 % Variance is moderately inflated by outliers ```

I maintain dependencies for a lot of people--nowhere near operating systems or stdlibs, but my OSS gets fairly broad use--and I try very hard not to change their APIs too much! It's a real concern.

Nope! For a few reasons Jepsen sticks very closely to Traditional JVM Threads.

Part of that is because Jepsen isn't hyper-concurrent. Most concurrency errors manifest in just a handful of client threads, and the JVM will run ~10K threads without too many issues.

Another part is that I spend a lot of time debugging and profiling. Core.async turns stacktraces into Swiss cheese--it has to, to perform its implicit continuation-passing magic. But that can make it really hard to figure out Who Called What When, and Why Is This Process Slow. Standard blocking IO calls are also something that my profiler (YourKit) can analyze well.

Finally, I invest a lot of time into tuning Jepsen for speed, and manage the handoff of data between threads carefully. In a system where actors were handing off state constantly core.async would be a bigger help, but there's really only a couple shapes for concurrent handoff in Jepsen, and they're addressed by either j.u.c. queues, forkjoin-style structured concurrency, or Weird Transactional Threadpool stuff.

For a little more on this, take a look at https://github.com/aphyr/dom-top or https://github.com/jepsen-io/history/blob/main/src/jepsen/hi...

Tbh I think they've actually gotten worse in recent years. This thing where the compiler spits out a stacktrace to /tmp and makes you go and cat it? Absolutely awful.

And the move to spec, ugh, UGH. I have wasted so much of my life looking at spec errors for (e.g.) namespace declarations and being completely unable to tell what was wrong. I don't know how to fix this, unfortunately.

Speaking very loosely, primitives on the JVM are values which are represented directly in memory, instead of as pointers to objects on the heap. Clojure (again, very loosely) generally treats everything as a pointer to a heap object. There is no specialized equivalent for, say, a vector of shorts, or a map where values are floats. The compiler can emit specialized function signatures for... IIRC longs and doubles, but other types (e.g. byte, float) aren't directly accessible--they go through widening conversion. It's also easy for the compiler to quietly fail to recognize it can preserve primitives in some kinds of loops, so you wind up with what Java calls "autoboxing": wrapping a primitive in a corresponding Object type.

Here's a recent example of some code in a hot path inside Elle, one of Jepsen's safety checkers. It does a lot in primitives, using packed structures and bitmasks to avoid pointer chasing.

https://github.com/jepsen-io/elle/blob/main/src/elle/BFSPath...

There was actually a Clojure version of this earlier that got pretty close perf-wise, but I wound up dropping to Java for it instead:

https://github.com/jepsen-io/elle/blob/913cbff5ebb19ba850c0a...

There's a happy medium, I think, and I also imagine that depends on scope of change and the kind of applications involved. Speaking personally, I'm OK rewriting code, say, every decade. In the Ruby world, things moved so fast that my code would rot every six months. Now that I've been maintaining the same software for 20 years I'm really starting to care about that. ;-)

Clojure's Java interop story is generally really good. Method calls are predictable and uniform. The collection types generally work both ways, rather than being a completely independent type hierarchy (as a former Scala programmer, what a breath of fresh air). Functions work as Runnable, Callable, Comparator, etc. Reify, deftype, and proxy are sufficient for maybe 98% of APIs I've worked with.

There are, however, those rare exceptions! For instance, some Java APIs literally won't work without annotations. There's just no way (last I checked) to express those in Clojure.

I don't remember exactly the nature of this problem, but I've had occasional issues with deeply overloaded argument dispatch before. Maybe it was something like "The Java API expects an object x which responds to x.foo(long l) and also x.foo(int i) differently", and there was no way to convince deftype to emit the right kind of class.

AOT is... it's weird. You want ahead-of-time compilation when, say, you want to write a Java class that uses datatypes defined in Clojure. But AOT, at least in Leiningen, tends to break stuff in truly inscrutable ways. My professional opinion is that it's haunted.

Clojure has a gradual type system--core.typed--that I've used extensively. It's... it's very cool, but for my uses it seems to be more trouble than it's worth. I keep coming back to it every few years, though.

DB vendors generally don't write Haskell clients, which put a serious damper on my "test lots of databases for a living" project. Love Haskell as a language, but it's just not a popular target, and I wasn't brave enough to go for C interop. As I touched on in the post, Haskell is also rigorous about side effects--I've found it difficult to, say, inject side-effectual performance monitoring into an otherwise pure piece of code so I can profile it. Maybe Haskell experts are better at this, or the story is better now.

There was a Ruby version of Jepsen too. That enjoyed better library support and language flexibility, but it suffered from terrible performance, a standard library built for mutability, and a janky thread model. There was, at the time, essentially no way to reliably time out a Ruby thread. That was a dealbreaker all in itself.

I wrote some of Jepsen in Erlang too, but abandoned the project quickly. It's got a really good concurrency story. A VM designed for concurrency and failure with bounded reduction queues--honestly this is something I miss in the JVM. I love Erlang as a language (and I think some of my Erlang might still be in prod!) but it also suffered from a lack of client support, not being well-suited for high-speed mutable state, and generally terrible string manipulation. There's a lot of strings in Jepsen. ;-)