See https://github.com/codership/galera/issues/336#issuecomment-... (linked from the article)
There will be bugs but in the end if your goal is to write a consistent store, you can do it. Bugs in wanna-be CP systems and/or failed attempts at implementing CP because of home-made protocols with flaws are part of the issue and part of what Aphyr's research uncovers, but IMHO the central point is a different one: CP systems have performance limits so many real world systems don't even try to be CP, which is fine, it's up to the designer to pick the DB design and tradeoffs, but you need to document it properly. What happens during partitions? What consistency model the system employs if it does not feature strong consistency? Is the system Available during partitions? And so forth. As long as the documentation is honest, it's up to the user to understand, for its use case, if the system is a good fit or not.
What we are seeing often is an attempt to informally document what should be well specified. Sometimes there are also incorrect things stated in bold letters in the documentation, like "the system is consistent but sacrifices partition tolerance" which does not mean anything in the context of partitions actually happening in the real world.
Aphyr's effort and our collective effort as people working, implementing and using databases, in my opinion, is not to end with databases designed all the same way or all providing strong guarantees, but with clear understandable and honest documentation describing the system behavior.
They try on paper, so to speak. They might have a CP component, but it is not in the datapath -- it is there for maybe cluster configuration, or say to elect a leader. And then they can claim they have CP because everything goes through the leader, and the leader was elected by a CP component. However without realizing they just pushed all the error cases into corners -- when the membership changes (nodes die, die in unpredictable ways, become partitioned, new nodes added...). They never test that extensively, which is a hard thing to do, because there are so many cases and combinations.
The bottom line is even if they have a Paxos or Raft component in their system doesn't mean their data is stored consistently.
Reinforces that the hardest part in engineering is rarely the technical problem. Distributed databases are really f*ing hard, but infinitely harder if the people can't work together or don't open up to faults.
In part it's hard to convince everyone of the importance of spending time on what looks like an unlikely corner case until after the outage - when it also becomes much harder to fix as you usually need to build this into the core design of your system.
It's not hard — it's slow. Sending reads and writes through Paxos or Raft gives you sequential consistency. But not surprisingly, touching a quorum of nodes for every operation is too slow to be practical for many workloads.
And that's usually fine — most data aren't bank account balances.
Online transaction processing creates "pending" transactions, and the data is often inconsistent. Your charge may exist in the merchant's database but not post to your online banking for several hours. Or it may be a wildly different amount - i.e. gas stations will place a $100 hold on your debit card and it will stay that way for days until it's settled for the actual (lesser) amount. If you were accidentally double-charged, than rather than processing a separate refund, the merchant may simply not settle the duplicate charge, and it will drop off your pending transactions... eventually. The lag time may be several days or weeks. If it's a debit card, you can't spend the money during that time and you may be temporarily broke because of it.
If you make an ACH transfer, money will disappear from your account one night, spend a business day in the aether, and then post to the recipient's account on the third day. The system is in inconsistent state (i.e. money is missing) for at least a day, possibly a whole weekend.
The actual transaction is settled and goes into "posted" state with a lag of 2.5 * 10^8 ms - i.e. 3 business days. That's if you're lucky. Banks do need strong consistency, but not in anything approaching realtime.
Even ZooKeeper could probably handle the U.S.'s financial transactions faster than current infrastructure.
Probably not even going via ACH, rather through treasury department nettings, mitigated against each other, just because that's accepted.
RTGS really is Real Time Gross Settlement. If you're led to believe elsewise, you're being conned.
What might be considered an appropriate data store for one data set may not be for another. And the type of information a lot of distributed systems are handling are far from bank accounts, for example.
And if you're handling ultra important, sensitive data there are techniques that have been available for many years (two phase commit, for example) that can help.
I love this series but I'm blown away by how many engineers here automatically assume a system is a failure because it doesn't pass a certain type of test from time to time. I do agree with the series that the marketing materials shouldn't claim things, however. Companies should be honest with what their systems weaknesses currently are.
I suspect that no experienced programmer expects to get data safety for free. We do expect that the documentation doesn't lie to us when describing the level of data safety provided.