irfansharif

If you're just using the pre-compiled binaries, this Go patch is baked in. If you're building the CockroachDB binary from source, and are just following our usual build steps using Bazel, you'll get this stuff as is. There's no need to compile a custom Go first.

(crdb eng) I'm not sure what "document DB" means here, mind elaborating?

Yes, the distributed backup/restore functionality was folded into the free version in our last release. See https://www.cockroachlabs.com/blog/distributed-backup-restor....

(crdb eng here) I'll shamelessly call out that distributed backup/restore, which was previously in our enterprise offering, was moved into the free one. Internally a large part of that decisions was driven by the "suggestions" made here on HN, so keep 'em coming.

Click around through the rest of their blog if you're interested in what those tradeoffs are. Let's not spread FUD unnecessarily.

It's not, not at all. We've actually gone as far as replacing RocksDB with https://github.com/cockroachdb/pebble/ by default now (and it's written in Go).

what "data warehousing"/"data management" tools are missing?

> Note that neither CockroachDB [...] use Golang for their actual storage engine

We do, now. We're looking to move away from RocksDB to https://github.com/cockroachdb/pebble/.

Congrats on the launch! If you're looking at CockroachDB (looking at some other comment on this thread), you should reach out to us if you haven't already. Your design here seems like the exact kind of thing we're hoping to push towards.

(Linked in the post but) github repo: https://github.com/MaterializeInc/materialize

I have no idea what you're saying. What's a "row-oriented storage" if not storing all the column values of a row in sequence, in contrast to storing all the values in a column across the table in sequence (aka "column store"). What does the fact that it's exposed behind a KV interface have to do with anything? What's "more" about Postgres' "row-orientedness" compare to MySQL?

In case you didn't know, a row [idA, valA, valB, valC] is not stored as [idA: [valA, valB, valC]]. It's more [id/colA: valA, id/colB: valB, id/colC: valC] (modulo caveats around what we call column families[0], where you can have it be more like option (a) if you want). My explanation here is pretty bad, but [1][2] go into more details.

[0]: https://www.cockroachlabs.com/docs/stable/column-families.ht...

[1]: https://www.cockroachlabs.com/blog/sql-in-cockroachdb-mappin...

[2]: https://github.com/cockroachdb/cockroach/blob/master/docs/te...

> Actually, I always believed that the internal key-value store that they use would never scale to represent table workloads.

Care to elaborate here? As someone working at that layer of the system, our RocksDB usage is but a blip in any execution trace (as it should be, any network overhead you have given it's a distributed system would dominate single-node key-value perf). That aside, plenty of RDBMS systems are designed such that they sit atop internal key-value stores. See MySQL+InnoDB[0], or MySQL+RocksDB[1] used at facebook.

[1]: https://en.wikipedia.org/wiki/InnoDB

[0]: https://myrocks.io/

> Just, please, pretty please, give us a --insecure

Have you taken a look at [0]? (All caveats around running an insecure cluster apply)

[0]: https://www.cockroachlabs.com/docs/stable/deploy-cockroachdb...

TrueTime is not a publicly available GCP service.

We take our claims if correctness very seriously. The behavior you're describing, depending on exactly what you're doing, is likely an allowed serializable history, which is the isolation level we claim to support. See [0] for a deeper dive on the subject. Either way, you should show us what you found (if you haven't already).

As for your performance analysis, do share your results and methodology. We published an in-depth, reproducible comparison with YugaByte here [1] in addition to publishing our TPCC-100k numbers[2]. If you're seeing performance that doesn't line up with the above, let us know.

[0]: https://www.cockroachlabs.com/blog/consistency-model/

[1]: https://www.cockroachlabs.com/blog/unpacking-competitive-ben...

[2]: https://www.cockroachlabs.com/blog/tpcc-100k/

CockroachCloud has public pricing. https://www.cockroachlabs.com/docs/cockroachcloud/stable/coc...

As someone working for crdb, I disagree with you re: "the whole point is that it enables Google scale". There's a wide gulf up to that point that cockroachdb is still imminently useful for, especially considering the alternatives.

CockroachDB runs the Jepsen test suite nightly. We've been following along Aphyr's recent test additions (`multi-register` for instance, which immediately caught [0]), porting them over when appropriate. We definitely have work to be done incorporating the more DDL focused tests that tripped up YugaByte.

[0]: https://github.com/cockroachdb/cockroach/pull/40600

v20.1 is slated for release in April.

Those numbers are surprising to me. Have latency charts to share? And are you using think time? http://www.tpc.org/tpcc/detail.asp

> cdb went with Go but their storage layer ended up being rewritten with C,

The storage layer was not rewritten in C, it's just RocksDB. There's ongoing work to use a custom built LSM store instead: https://github.com/cockroachdb/pebble

The core shopify codebase is a monolithic one, hence the merging infrastructure described in the post. It's not a "single project" in any useful definition of the phrase.

Not technically open-source (BSL-licensed), but yes, source available on https://github.com/cockroachdb/cockroach/.

You can find our more recent performance numbers on https://www.cockroachlabs.com/docs/v19.2/performance.html

Because we blogged about our new vectorized sql engine[1] & our new parallel commit protocol[2] recently, two "big"-ish engineering lifts that were packed into this one release.

[1]: https://www.cockroachlabs.com/blog/how-we-built-a-vectorized...

[2]: https://www.cockroachlabs.com/blog/parallel-commits

I'll repeat what was mentioned elsewhere. The GC pauses in Go are in the order of microseconds. We've done a ton of work keeping our memory usage under tight bounds and keeping a close eye on exactly how the go GC processes interplay with query execution threads (what processors are consumed, GC pressure buildup, where/when the stalls are introduced, etc.) Consistently we've found that given you're invariably dealing with network latencies (it's a distributed db after all), the microsecond GC pauses are almost entirely dwarfed to matter (I'm not excluding p99s here).

Agreed, we should be talking about the "timestamp cache" in more detail generally. While I'm here, looking at [1] helped me confirm how everything is kosher despite being in-memory. The timestamp cache pessimistically maintains a "low water mark", this always ratchets up monotonically and represents the earliest access timestamp of any key in the range. Writes happening at timestamps lower than this water mark are not let through, and bumping this watermark past the point of the observed write intent's timestamp is how slow inflight write intents are aborted by recovering read requests. On server restart, the timestamp cache is initialized with a low water mark of the current system time + maximum clock offset, so "future" write intents (or more accurately: write intents sent in the past but previously stuck in transit) are simply rejected.

[1]: https://github.com/cockroachdb/cockroach/blob/master/pkg/sto...

> I see now that it only applies in the after phase of transactions without a failure.

Yep.

> Would that scanning, analogous to RAID "resilvering" subject to write-intent ranges to limit the keyspace regions scanned, usefully improve read latencies later?

I think it's just a better design to have it done on demand. The keyspace is large and failures are rare, and when one of these zombie intents are happened upon, the very first reader addressing it resolves it for all subsequent readers. A global scan would improve read latencies later, but not by much and not for many readers.

> If I understood correctly, the extra round trip on the reader only occurs with left-over intents, which are the product of an earlier failure.

Left over intents are also visible for an ongoing txn, before those intents are resolved. But there's no added latency in the read path, I've commented elsewhere in the thread to explain how.

> After coordinator failure events, the new coordinator starts to clean up left-over intents asynchronously (nothing specific is waiting for it to finish).

The cleanup happens by readers on demand, there's no separate global coordinator scanning the keyspace and resolving old write intents.

> Though I understand it differently, as reader just waits until all writes and "txn COMMITTED" record arrive at it's node.

Actually, welp, I believe this is closer to the truth in implementation. But consider how read performance compares before and after introduction of Parallel Commits. Before, when readers happen over extant 2PC "prepare" phase markers, they would still have to wait for txn resolution (on the coordinator node of the other txn, or on the node the intent was seen). They simply continue doing the same in Parallel Commits, there's no extra latency added to the read path (except when there is failure, but even then as soon as the earlier txn is recovered, future readers no longer get stuck).