Hacker News Reader

Comment by void_star | Hacker News Reader

It's a little unfortunate that consistency is only mentioned as a footnote. If this is being used internally at Netflix, how are they dealing with consistency issues in the face of failed or conflicting writes?

View on HN

This currently looks like the Dynamo-alike sharding without the read-repair part making values eventually convergent in all the replicas. This latter feature is much needed, but they are already planning it apparently.

Except no mention of vector clocks, so it's unclear how one would detect or repair a conflict. Maybe that is an implementation detail that is intended to be ironed out later...

With a replication factor of 3 or more you don't need a vector clock, you can just choose majority wins.

That being said, I believe it is using a vector clock, but I'm not sure.

You still need ordering amongst writes. Without that you will get into a situation where there is no majority, all three nodes have different data.

You need quorum writes and reads. Like Cassandra you can use timestamps even though vector clocks would be much preferred.

Converging to a single value (which is a liveness property) and what the values will contain (which are usually safety properties) are different stories... when the partition heals, if the nodes detect they have a different version for a given piece of data, they can just remove it, and the system is convergent, which means it is eventually consistent (a liveness property). What happens in the merge function instead provides additional safety properties. For example in the case of a CRDT that models a Set type, with the merge function that is "union of all the values in the nodes", you can state the safety property that if a Set element X was added into a node, at no time will happen that the nodes will be able to talk again, and the system will converge into a Set which excludes such value X.

Vector clocks are not mandatory, depends on what you want to build. There are different options like CRDTs or using a merge function which guarantees a weaker form of consistency.

If it's based on Dynamo, then it's using vector clocks.

Reply on news.ycombinator.com