Hacker News Reader

I believe you're right. They go into their design in their MySQL HA post:

https://githubengineering.com/mysql-high-availability-at-git...

> In MySQL’s semi-synchronous replication a master does not acknowledge a transaction commit until the change is known to have shipped to one or more replicas. It provides a way to achieve lossless failovers: any change applied on the master is either applied or waiting to be applied on one of the replicas.

They only require one other replica (as opposed to a quorum) to be reachable from the master for the master to continue acknowledging writes. If a new master has been elected on the other side of the partition, both will continue acknowledging writes.

They noted this as a limitation that they were working on (but unfortunately a bit too late in hindsight):

> Notably, on a data center isolation scenario, and assuming a master is in the isolated DC, apps in that DC are still able to write to the master. This may result in state inconsistency once network is brought back up. We are working to mitigate this split-brain by implementing a reliable STONITH from within the very isolated DC. As before, some time will pass before bringing down the master, and there could be a short period of split-brain. The operational cost of avoiding split-brains altogether is very high.

I only skimmed that post, and havent dealt with mysql in years, but .... that bit has another wiggle with writes committed to one of many replicas. Im presuming thats any one of many. Which means I can now have writes committed not just in two places (terrible), but to N number of places (zomg). Recovering that (linearly) would tend towards impossible without something like a vector clock. But if you had a reliable clock of changes you probably wouldnt be in that mess at all.

You don't need vector clocks for master/slave replication, even if there's 1 master and N slaves.

For consistency, all of the slaves should be applying the stream of updates to their local replicas in the same order that they happened on the master. That means each slave has seen a prefix of the master's data; its state can be uniquely represented by a position in the replication log. So you can easily determine which replica is most up-to-date by taking the one with the largest position. (If you're doing synchronous replication correctly, at least one replica is guaranteed to be completely up-to-date, even after an unexpected failure.)

Vector clocks only come into play when you are simultaneously accepting writes on multiple masters, and you need to keep track of which updates have already been applied from each source.

> If you're doing synchronous replication correctly, at least one replica is guaranteed to be completely up-to-date, even after an unexpected failure.

Exactly? Maybe? Im reading a lot in to little information here. It wasnt clear to me if "one or more" was the master + a single specific replica, or something closer to master + 1/N. And then there's the automated optimistic leader election. Combine those with (multiple) partitions and you could have a very bad day.

Rereading their RCA now I'm guessing the above isnt the case. "Restore the entire datacenter" reads more like they preserved the ~40 seconds of east coast partition writes, restored everything in east to a known checkpoint, and resumed replication off the more advanced masters in the west coast. If it's "just" two masters per cluster, both of which are available and internally consistent, you "just" have to reconcile those two write streams.

I'm by no means an expert on any of this stuff, but:

> They only require one other replica (as opposed to a quorum) to be reachable from the master for the master to continue acknowledging writes (from your comment)

> Orchestrator considers a number of variables during this process and is built on top of Raft for consensus... (from the article)

Doesn't quite make sense to me. Doesn't raft require the master to wait for a quorum before committing writes? I understood it as a pretty important aspect.

> A candidate must contact a majority of the cluster in order to be elected, which means that every committed entry must be present in at least one of those servers (from the raft paper [1])

I understood that the bold is only true if commits are acknowledged by the quorum

Edit: I'm not implying your post is incorrect, just trying to understand how the two fit together

[1]: https://raft.github.io/raft.pdf

This was my first thought, you beat me to it. It seems like they contradict each other. Is it Raft with quorum or is it a single replica node that is caught up with master? You can't have both (unless you only have three nodes).

> unless you only have three nodes

Interestingly enough this seems to be how managed sql solutions do it. You just have one primary and one failover replica in another zone. So a quorum write is just a synchronous write to both (2 out of 2). You don't have the split-brain problem because the original primary can't make progress if it can't contact the replica.

https://cloud.google.com/sql/docs/mysql/high-availability

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Conce...

Github's problem is that they were trying to be too smart and allow any of their replicas to be candidate masters without increasing their quorum size. In theory this is higher availability than the managed sql solutions (they can be available even if their entire coast gets nuked) but they do it at the cost of consistency in the more common failure scenarios.

> Doesn't quite make sense to me. Doesn't raft require the master to wait for a quorum before committing writes? I understood it as a pretty important aspect.

Different kinds of masters. I've forced myself to always include the application if terminology gets reused across applications in a discussion. As far as I understand this>

- The mysql-master only commits an sql-level-transaction if a mysql-replica acknowledges replication of the transaction. This avoids data loss during a failover.

- The orchestrator-nodes elect an orchestrator-master via raft mechanics.

- After this, the orchestrator-master starts pondering about the mysql-master / mysql-replica situation and potentially promotes the most current mysql-replica to mysql-master.

- This promotion requires writes on an orchestrator/raft level, which in turn requires the orchestrator-master to get an acknowledgement from the orchestrator-quorum.

Github chose availability over consistency. Bad choice?

I would say yes. The service was available but it was even more chaotic because of all the inconsistencies.

It sounds as if a switch was replaced so network connectivity was lost for 40 seconds (a partition if you will). The network recovered but during the 40 seconds a lot of "things" had to queued and processed and things cascaded out of control. Whenever people talk about CAP they seem to spend time thinking about behavior when there is a partition, and that's neat an all, but the real challenging parts come when the partition is healed, and people don't seem to spend enough time thinking about that mode of execution.

Or when there isn’t a clear/complete partition and the network is kinda working but not consistently.

> But just based on what was explained in this blog post, it sounds like they're replicating database updates asynchronously, without waiting for the updates to be acknowledged by slaves before the master allows them to commit. Which means the data on slaves is always slightly out-of-date...

Anecdotally I think I see this regularly. Part of the build pipeline I work on requires recursively downloading a particular folder for a given repo. From time to time, I'll have the latest git ref give me data from the previous git ref. Everywhere else on GitHub the information is up-to-date. But not the API endpoint to download files. When this happens it takes roughly 5-7 minutes for the correct information to show up.

After reading this and all the other resources people have posted I'm definitely going to see if I can reliably reproduce the error.

Why would you use an API endpoint to download files in a build pipeline? Surely you should just use a clone of the repository?

I could believe that an endpoint to download files looks at a cache, and if you are just using 'master' or 'latest' as opposed to a commit, I could believe that would be out of date.

Bit late, but replying to this because my comment does have missing context. Most of the repos we have are large enough that downloading three individual files is faster than cloning the whole repo. Even with a shallow clone with a depth of 1. I've tried the combinations and even tried downloading a zip file of the repo. Still faster to download the three files. And I haven't even added parallel downloads because it's fine as is.

But given the issues, why continue to use the API? Because those files are related to the build pipeline logic and they change maybe once in 3-4 months. I can live with an error once in 3-4 months. Recently I was working on changing some of the logic of the pipeline and that's when I ran into it enough to say that there was something odd from Github's end.

"Which means that promoting a slave to master will by definition lose some committed writes."

Sure. But that is where the application code comes in. You usually have at least two db connections one to the slave connection and one to master. If you are checking if an account exists [important] you can ask the master since it will be in an up to date state. If you want to build someone's feed on github, then doing all your sql though the slave is fine. They also probably have layers of atomic cache on top which handles 99% of the requests [memcache,redis,whatever]. EG. Does the account exist? Check the cache, check master.

Restating what I think you're suggesting:

Use master for transactional read/writes.

Use replicas (slaves) for readonly.

More or less?

I'm sure that is what they do. But application code does not save you from this scenario. If your master changes, the application does not know that the new master was a node that did not have full replication.

Typical solution is to mix synchronous block level replication intra-DC with asychronous replication inter-DC. I believe RDS utilizes DRBD for this intra-region.

This may be why failover is "safe" in certain circumstances.

> it sounds like they're replicating database updates asynchronously, without waiting for the updates to be acknowledged by slaves before the master allows them to commit. Which means the data on slaves is always slightly out-of-date, and becomes more out-of-date when the slaves are partitioned from the master. Which means that promoting a slave to master will by definition lose some committed writes.

Unless you temporarily pause writes to the primary before promting a slave.

Sure, that's fine if you're talking about planned downtime. You still lose data if there's an unexpected failure, because by the time you realize the primary is gone, it's too late; the writes have already happened and been acknowledged.