HNHacker News
TopNewBestAskShowJobs

irfansharif

1,122 karma · joined June 14, 2016

irfansharif.io/blog
submissionscomments
irfansharif··on Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions
> Am I understanding correctly that readers' performance will degrade by the same amount?

Not quite. The "slow path" talked about is only applicable when the coordinator node is unavailable (presumably a rare event). If it's unavailable, there's nobody left to clean up the STAGING txn record, so the reader is tasked to do it itself.

In normal conditions however, once the coordinator node receives acknowledgement for the successful persisting of all write intents and the "txn STAGING" record, it can simply record "txn COMMITTED" in memory (and return to the client, send off async intent resolution procedures, etc.) Any subsequent read requests that observes left over intents (yet to be resolved) are pointed to the coordinator node, which can simply consult the "txn COMMITTED" record in memory. This is all safe because the commit marker is not simply stored on the coordinator node, it's a distributed condition and can be reconstructed by any observer even if the coordinator failed.

irfansharif··on Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions
So if I'm reading this right, the master process can lose writes (if there's permanent failure). So taking your 2PC scheme above with the 'settled' flag, you can respond to the client that the txn has been committed once you've written the flag, but this commit marker can also just be entirely lost? (again, permanent failure)

If this 'settled' flag exists on each 'shard' instead of just the coordinator, any random subset of those too can be lost? I don't understand what's going on here, or what guarantees this 2PC implementation provides.

irfansharif··on Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions
The "criticism" as it pertains to CockroachDB and off-the-shelf 2PC is less so about 2PC in isolation and more so about layering 2PC on top of consensus groups used to persist records. When any given txn does the "prepare" phase, it lays down markers for a possible upcoming commit. If the 2PC coordinator fails in an inopportune moment, there's a delay between the failure and the markers being cleaned up (whether or not the transaction is "reprocessed" or aborted). The reason why this delay is problematic is because any subsequent transactions that happen upon said markers, they just have to wait for the resolution ("commit"/"aborted") aka it blocks. So clearly recovery must be built into 2PC, i.e. the transaction state itself must be persisted. This is done so in the same way the markers/regular writes are, through consensus. But marking the transaction state as "committed" can only happen once we're guaranteed that all the individual write markers are persisted. Which adds a second round of consensus.
irfansharif··on Re-Licensing Sentry
> Although we’ve come to refer to the BSL as eventually open-source since it converts to an OSI-approved license at the conversion date, due to the grant restriction, it is formally not an open-source license.

They're very clear in stating that the license is not open source.

irfansharif··on Go 1.11 Rc1 released
From Go's 1.0 compatibility guarantee[0]:

> It is intended that programs written to the Go 1 specification will continue to compile and run correctly, unchanged, over the lifetime of that specification. [...] Go programs that work today should continue to work even as future "point" releases of Go 1 arise (Go 1.1, Go 1.2, etc.).

I've found this to be upheld in practice, especially so for the standard library. I have yet to come across a case where I was concerned with which "version" of the standard library some project was written against. Are you perhaps talking about non-stdlib packages?

[0]: https://golang.org/doc/go1compat

irfansharif··on CockroachDB 2.0 released
For those in the area, the CockroachDB folk are hosting a meetup demoing some of the new 2.0 niceties: https://twitter.com/CockroachDB/status/978701442050592768?s=...
irfansharif··on VeritasDB: High Throughput Key-Value Store with Integrity
File system.
irfansharif··on Open-sourcing a 10x reduction in Apache Cassandra tail latency
RocksDB is also an LSM structured KV store.
irfansharif··on Distributed Databases Should Work More Like CDNs
No? Apologies if I'm not able to communicate as clearly. The RTT would be to the furthest replica within `n` replicas ordered by proximity to the query origin, where `n` is the number it would take to have a majority within the specified replication factor (so 2 for a replication factor of 3, 3 for 5, etc.). It would be different from the RTT observed with Amazon RDS as long as the furthest replica, as described above, is in a different zone/region/DC as compared to RDS's primary instance (assuming a similar topology). So with a replication factor of 3, if your DCs are in the UK, Australia and the US, if the RDS primary was in the US, all write requests from Australia would have to travel over to the US. But with a replica in each zone, write requests from Australia would hit the DC in Australia, and the DC in UK, avoiding the larger RTT to the US.

To reiterate: I believe the argument is that this a more flexible design point than a single point of entry for all incoming writes, regardless of the origin.

irfansharif··on Distributed Databases Should Work More Like CDNs
Given the flexibility of where the range raft leader could be, CRDB makes an active effort to colocate it near to where the requests originate from (which is some part of what the CDN parallel was alluding to with low RTT for multi-region deployments).

WRT to the writes here, if a majority of the replicas for that range are in the proximate regions, the requests would only travel that far before responding. I believe the argument is that this a more flexible design point than a single point of entry for all incoming writes, regardless of the origin. The cost to write out to the furthest region within any majority of replicas is of course inevitable to have cross-region durability, alternatively you could trade this off to have the majority of your replicas specific to requests from a specific region, be located to that specific region.

irfansharif··on Distributed Databases Should Work More Like CDNs
At the key-value level, CockroachDB starts off with a single, empty range (a set of sorted, contiguous data from your cluster). As you put data in, this single range eventually reaches a threshold size (64MB by default). When that happens, the data splits into two ranges, each again covering a contiguous segment of the entire key-value space. This process continues indefinitely; as new data flows in, existing ranges continue to split into new ranges, aiming to keep a relatively small and consistent range size. Each range is replicated 3-way (by default) as well, and is backed by a single Raft instance.

When your cluster spans multiple nodes (physical machines, virtual machines, or containers), newly split ranges (or more specifically, replicas of these ranges) are automatically rebalanced to nodes with more capacity. Writes addressed to a range are handled by the Raft leader for that range (which can hop around its various replicas as needed). Writes to different ranges (non-overlapping key spaces by definition) are processed independently, and very well may be processed across multiple machines.

Source: https://www.cockroachlabs.com/docs/stable/frequently-asked-q...

irfansharif··on Distributed Databases Should Work More Like CDNs
I doubt this particular article was intended to address how CRDB handles writes cross-region, synchronous replication, etc. You'll find articles touching on varying aspects of what you're looking for if you dig through some of the earlier posts on their blog[1] or their FAQ[2].

[1]: https://www.cockroachlabs.com/blog

[2]: https://www.cockroachlabs.com/docs/stable/frequently-asked-q...

irfansharif··on Decimal: A high-performance, arbitrary-precision, floating-point decimal library
As a plug for another arbitrary-precision, floating-point decimal library in Go, there's https://github.com/cockroachdb/apd.
irfansharif··on Why Go and Rust Are Not Competitors (2015)
Not sure I quite understand the premise that there is a strict delineation between programmers who are "prepared to accept more complex syntax and semantics for <...>" and otherwise. I can think of a lot of cases where I'd personally use either or (I've been looking into tikv contrasting it with cockroachdb's storage layer for e.g.), a lot of cases where I've used Rust's formalization of ownership/borrowing semantics to alleviate GC pressure in Go code. I'm hard pressed to find projects "for" Go that Rust wouldn't be a reasonable alternative to, and vice-versa.
irfansharif··on Executing Parallel Statements to Improve Performance
The accompanying RFC, Parallel SQL Statement Execution: https://github.com/cockroachdb/cockroach/blob/0da3da734ce6bf...
irfansharif··on CockroachDB 1.1 Released: Production Made Easy
> Kyle’s testing found two new bugs: one in CockroachDB’s timestamp cache which could allow inconsistencies whenever two transactions are assigned the same timestamp, and one in which certain transactions could be applied twice due to internal retries. Both of these issues were fixed.

Source: https://www.cockroachlabs.com/blog/cockroachdb-beta-passes-j...

If you're speaking to "Linearizability violations such as stale reads can occur when the clock offset exceeds CockroachDB’s threshold." from section 2.1 in Aphyr's analysis[0], this too is touched upon in the post above:

> loose clock synchronization is necessary [...] to guarantee serializability. CockroachDB servers monitor the clock offset between them [...] although this monitoring is imperfect and may not be able to react quickly enough to large clock jumps. All CockroachDB deployments should use NTP to synchronize their system clocks [...] the default of 500ms (increased since Kyle’s testing, when it was 250ms) to be reasonable in most environments, including virtualized cloud platforms.

[0]: https://jepsen.io/analyses/cockroachdb-beta-20160829

irfansharif··on Cloud Firestore: A New Document Database for Apps
Congratulations on the release!

> multi-region replicated database [..] once data is committed, it's durable [...]

> strongly consistent on the server-side

Do you mind elaborating a bit more here? Around perhaps what happens underneath the hood when failing over, etc. Do you have a single "co-ordinator" of sorts ensuring strict serializability, if so what do you do when failing this over? Or is it a quorum based approach like Paxos/Raft?

irfansharif··on Interesting Codebases
There was some discussion on the very same for an Ask HN[0]. Copying my comment from there I've found the google/leveldb[1] source code to be immensely educational, authored by Jeff Dean and Sanjay Ghemawat. The implementation of leveldb is similar in spirit to the representation of a single Bigtable tablet[2].

[0]: https://news.ycombinator.com/item?id=13854431

[1]: https://github.com/google/leveldb

[2]: http://research.google.com/archive/bigtable.html, section 5.3

irfansharif··on Go Assembly by Example
Tangentially related, recently came across this: https://go.godbolt.org/

It's an interactive compiler allowing you fiddle around with the generated assembly.

irfansharif··on Ask HN: What is your favorite CS paper?
Diego Ongaro's Raft paper[1]. Perhaps this only speaks to my experience as a student but having surveyed some of the other papers in the domain (paxos[2] in its many variants: generalized paxos[3], fpaxos[4], epaxos[5], qleases[6]), I'm glad the author expended the effort he did in making Raft as understandable (relatively) as it is.

[1]: https://raft.github.io/raft.pdf

[2]: https://www.microsoft.com/en-us/research/wp-content/uploads/...

[3]: https://www.microsoft.com/en-us/research/wp-content/uploads/...

[4]: https://www.microsoft.com/en-us/research/wp-content/uploads/...

[5]: https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf

[6]: https://www.cs.cmu.edu/~dga/papers/leases-socc2014.pdf

irfansharif··on Don't Settle for Eventual Consistency (2014)
CockroachDB’s external consistency guarantee is by default only serializability. We do provide a limited form of linearizability however, providing ordering for any observer or chain of observers[1].

[1]: https://github.com/cockroachdb/cockroach/blob/master/docs/de...

irfansharif··on CAP: Don't settle for eventual consistency
We use Raft underneath the hood to maintain consistency across the replicas.

re: 'There is no GPS sync'd atomic clock in Cockroach', Cockroach actually has a command line flag (--linearizable) which makes it behave like Spanner: wait for the max clock offset before returning a successful commit[1]. This isn’t practical if you’re using NTP synchronization, but if you have atomic clocks at your disposal..

[1]: https://www.cockroachlabs.com/blog/living-without-atomic-clo...

irfansharif··on The Limits of the CAP Theorem
Here is what Google has to say about 'moving the "C" part up into your application':

“We also have a lot of experience with eventual consistency systems at Google. In all such systems, we find developers spend a significant fraction of their time building extremely complex and error-prone mechanisms to cope with eventual consistency and handle data that may be out of date. We think this is an unacceptable burden to place on developers and that consistency problems should be solved at the database level.”[1]

[1]: https://yokota.blog/2017/02/17/dont-settle-for-eventual-cons...

irfansharif··on Local and distributed query processing in CockroachDB
[cockroachdb engineer] Baidu and Heroic Labs are ones we publicly announced in our 1.0 release[1], stay tuned for more~

[1]: https://www.cockroachlabs.com/blog/cockroachdb-1-0-release/

irfansharif··on CockroachDB 1.0
raft is premised on overlapping majorities, so to speak. in order to tolerate up to `n` node failures you'd need to run `2n + 1` instances (for nine nodes you'd tolerate up to four node failures).
irfansharif··on CockroachDB 1.0
> This is hard stuff but so cool too

I agree! we have some semblances of pushdown filtering across aggregations and some other interesting techniques as documented in the RFC[1] that first proposed the distributed execution model.

[1]: https://github.com/cockroachdb/cockroach/blob/master/docs/RF...

irfansharif··on CockroachDB 1.0
short answer: nope. cockroachdb replicates data for availability and in order to guarantee consistency across the replicas, it uses Raft[1] internally. Raft necessitates a majority of the replicas remain available in order to operate. it ensures that a new 'leader' for each group of replicas is elected if the former leader fails, so that transactions can continue and affected replicas can rejoin their group once they're back online.

[1]: https://raft.github.io/raft.pdf

irfansharif··on CockroachDB 1.0
haven't come across this idea before, interesting - will definitely have to give it some more thought. our 'distributed joins', so to speak, run through our distributed query execution model (distsql) setting up incremental 'stages' of computation with the results pipelined and plumbed through individual computes. viewing it through this model our implementation more closely resembles the Grace Hash Join[1] algorithm. you might be interested in the PR[2] that landed this changeset, there's a cool visualization in one of the comments[3] showing the query execution plan.

[1]: https://en.wikipedia.org/wiki/Hash_join#Grace_hash_join

[2]: https://github.com/cockroachdb/cockroach/pull/12221

[3]: https://github.com/cockroachdb/cockroach/pull/12221#issuecom...

irfansharif··on CockroachDB 1.0
the section on lock-free distributed transactions on our design document[1] should answer your question, specifically the sub-section on hybrid logical clocks.

[1]: https://github.com/cockroachdb/cockroach/blob/master/docs/de...

irfansharif··on CockroachDB 1.0
> I would be embarrassed to put "Cockroach Expert" on my resume

well, now that you mention it... https://irfansharif.io/irfan-sharif-resume.pdf

← PreviousPage 2 of 4Next →