Data consistency is overrated
two-wrongs.com
two-wrongs.com
There's consistency that's internal to a system. Do all of the foreign keys line up correctly? Have I lost any data that was provided to me? Here, we can aspire to be 100% correct. I don't think the examples in this article conflict with that.
Then there's consistency that's external to a system. This can be between this system and other systems, or between this system and reality. Did the operator enter the correct data for this item? Did the external system change and fail to tell me? Here there is no way to be 100% correct using only the tools that are inside the system. You need external audits, reality checks, periodic reconciliation.
Critically, all of the author's examples are about external consistency (accounting matching reality; inter-system communication), but their conclusion seems to be that because external consistency can't be fully achieved, we should be OK to abandon internal consistency within single systems. I think that's too strong a conclusion.
I'd argue that in your second case, there's no consistency "external" to the system; you've now defined a larger, distributed system that includes your external parties, and now you're in consistency-as-in-CAP territory, rather than consistency-as-in-ACID.
Great suggestion!
Specifically, for a large enough system, internal consistency will look more like external from a smaller system's perspective.
To that end, it is all about costs. If the cost of keeping consistent is not above the budget, do so.
As usual, balancing trade offs is better than following dogma, and within our current technological paradigm I don’t see any alternative but to minimize inconsistency for large distributed systems. Perhaps in the future we find something that eliminates a bunch of these problems altogether, but that’s not today.
In the beginning, there was less need for it - suuuper fast memory, but you might have a bit error on a pixel glitch for one frame and ... no big deal.
Then GPUs started to be used for compute and... well, we might not only need good memory, we might need ECC...
That's one of the key arguments that nonconsistency advocates are arguing for. If you build a music playlist, the behavior of the program if the mp3 files referenced within should be to skip that track, not to crash. I've heard too many people arguing that they should be allowed to emit nasal demons if there's a data error, and they're wrong even in many well structured and tightly integrated datasets, but especially wrong in "web-scale" datasets
"Nonconsistency advocacy" doesn't make a lot of sense to me here. Are you advocating not relying on consistency in systems that guarantee consistency? That is a waste of time.
Are you advocating eschewing consistent systems? Well, then you have more work, so only if I need to. And yes, you should handle data errors here because inconsistency is consistent with the system you've chosen.
This is true in a ton of cases where neither soft-delete or cascading delete make sense. Rarely, if ever, have I actually encountered the latter.
However, I'm not sure it's a place where you want to abandon foreign-key consistency. To display the song on the playlist (greyed-out, Spotify-style) you need its name and info, and you probably want to keep artist and album links valid as well. It sounds like this is actually a perfect use-case for a soft-delete: keep the song metadata in the songs table and mark it as nonplayable. This keeps the artist page, album page, and any links to the song itself valid, just greyed out when you get there.
- Strict consistency. Every pointer in my b-tree must point to a b-tree node, and not random data which could cause the program to crash. In a financial world, a bank should never print money.
- Fuzzy "good enough" consistency. In the examples in the article, all of the financial transactions should end up close enough to being reconciled.
There's value in both kinds of consistency. When talking about data entered by humans into a database, there's always going to be a bit of slop involved. Someone mistyped a digit. A few records weren't entered at all.
But when building software systems, designing with strict consistency guarantees is such an unbelievably massive win. I can't overstate how important it is. The entire ladder of abstraction in modern computers from transistors all the way up to this website is only possible because each layer "underneath" the layer we're standing on is solid and deterministic. If CPUs made even 1 error in every billion operations, our computers wouldn't boot at all.
Tony Hoare talks about his invention of null pointers as his "billion dollar mistake". Null pointers take something that should be strictly consistent (references) and make it fuzzy. Rust is exciting lots of people in the systems programming space because it takes things that are fuzzy in C (aliasing, memory management, thread-safe variables, etc) and makes them strictly consistent. All the value in unit testing comes from how they make our systems more strictly correct.
Every time I've relaxed consistency guarantees internally in systems I've worked on (or heard coworkers doing the same) we've come to regret it. At a startup several years ago, we needed to build an external search index for a database. The database updated live - and we had a change feed that updated the browsers live as records changed. The engineer in charge did a "good enough" job - he wrote a scrappy script that was only mostly correct. But it sometimes left the index inconsistent with the data. We got constant reports from our users about items not showing up in the search results. He would dutifully go back and fiddle with things to try and fix the problem. Eventually one of our senior engineers went in and rewrote the whole indexing script to be strictly correct. We never heard a peep about it after that - it just worked, every time. Even putting aside the frustration of our users, writing it correctly was a big win for us in terms of maintenance. Once it was correct, we didn't need to keep pulling engineering time away to fix problems.
Maybe data consistency is overrated in databases. But I think if anything, consistency internally in computing systems is underrated. We take for granted how well computers work. But our capacity to make computers do anything depends entirely on those consistency guarantees. It seems ridiculous to disregard its importance.
It's not relevant to the point but this is absolutely what banks do in the UK and I expect in most modern economies. They create money.
https://www.bankofengland.co.uk/explainers/how-is-money-crea...
Except banks routinely do "print money" (if you agree that credit and money are effectively the same). Even without delving into workings of our financial system, there are enough stories of banks making mistakes which erroneously increase their liabilities to clients. Banks simply revert such mistakes if they can or sue clients if money were withdrawn before such mistake was found. So it's much closer to fuzzy/eventual consistency.
I'd disagree; this is still strict consistency, at least in managed languages like Java or C#. (Coming from someone who has fully embraced nulls instead of fighting them.)
"Strict consistency" breaks in C and C++ where a pointer can either point to 1) a valid object, 2) be null, 3) point to an invalid address (i.e., dereferencing it crashes the program). Cases 1 and 2 can simply be distinguished by simple != 0 check. There is no (direct, simple, portable, defined by the language) check to distinguish between cases 1 and 3, which is where the inconsistency arises.
> I'd disagree; this is still strict consistency, at least in managed languages like Java or C#
The semantics are well defined, but the program that lives in my head (where the pointer is never null) can be subtly different from the program that runs on my computer. On the computer, the pointer can and will very occasionally be null, because I have a bug somewhere else in my code that the compiler isn't smart enough to point out to me.
That window of fuzzyness is wide enough to fit a peak hour traffic jam full of bugs.
> (Coming from someone who has fully embraced nulls instead of fighting them.)
Everyone who programs in C, Java, Javascript, C# and all the other languages with null pointers feel like they "embrace nulls rather than fight them". And yet, memory bugs still make up ~60% of the critical security vulnerabilities in Chrome and other systems software. I've seen so many crashes in application software over the years from null pointer exceptions (or the C/C++ equivalents).
I've been programming for 30 years and I still make occasional mistakes in these languages ending up with null pointer related errors.
I think its fair to say that moving away from implicitly nullable types lowers our collective defect rate. Language authors are on board too. Haskell, Swift, Rust, Typescript and even (I think) C++ now have non-nullable types in their type system.
Even if you feel like you're too smart to need the guard rails of non-nullable references, I'm definitely not that smart. And chances are, you run my code. Tightening up this stuff lowers the chance that my software crashes while you're using it.
- Option 1: nullable reference and if the program crashes, you have a bug. (That should be caught during testing.)
- Option 2: Maybe<T>, making the rest of the code unreadable. Because
- Option 2a: "Force-extract" the value and get another exception if the value is not initialized
- Option 2b: Do "proper" pattern-match and do ??? if the value is not initialized.
So I strongly prefer a plain nullable reference. If I dereference it, it raises an exception that gets handled at top-level _just like any other exception due to any other kind of bug_. There's no reason that, in a properly structured program, dereferencing null should crash the whole program. (In a managed language. Unmanaged languages have their own problems - e.g., impossible to determine if a non-null pointer is at all valid. Though OS _does_ let you handle segfaults, if you want to go that way. Windows structured exceptions are much more flexible there than anything available on unix.)
Part of this is guarantees about correctness. Is there a bug that manifests even without inconsistency, or was there just some weird inconsistent state that caused an issue? If you don't have consistency you have to ask this about every single bug, making debugging problems much harder.
My product managers would insist we do distinct counts on the aggregates instead of using probabilistic algorithms, because we "needed" the absolute 100% accurate output. No matter how many times I would explain the data was never 100% accurate to begin with and that the error rate using HyperLogLog wouldn't make an ounce of difference, we never were allowed to do that. As a consequence the performance of the system when doing distinct counts on interactive queries was about 100x worse. This didn't seem to bother anyone. I never understood why.
There are benefits to simplicity. If there isn't a reason to need 100x faster performance, then why complicate things? That would be my why.
But I want to add that anecdotally users do care about performance and they will often thank you for anything that is noticeable. Sometimes just because it feels nice, but it can also enable a fast feedback loop, which increases immersion and productivity.
In a broader sense, we use computers not only because they can perform work autonomously but also because they are fast, correct and remember details almost perfectly.
(Additionally, I would argue more generally that vote counts should be approximate. If a vote is close to tied then we run the largest risk of making an incorrect decision. Better to continue debate and/or refine the proposal until it has wider acceptance. This does not work for selecting government with term limits obviously.)
- They don't care about performance and think their users don't care about performance.
- They have sold something that doesn't make sense but sounds good.
- They are afraid of their users.
- Their users don't understand the problem you're describing or simply don't believe it.
- The business domain of their users actually knows the about the issue, but they still want or need the exact counts.
Everyone involved had roughly a decade of experience with the data in question, the processes that touched it before it reached us and within the company, the various problems with the data and how they impacted the analytics. They also understood how probabilistic algorithms would impact the results.
Additionally we had a team whose main responsibility was to continuously monitor and improve the quality of the data we were receiving. My team had frequent conversations with that team, because anything they couldn't detect would show up in our results.
They just had a fixation on "accuracy" stemming mostly from "our clients wouldn't understand it". Which is interesting because we had frequent complaints from clients who had secondary sources and found differences between what we provided and what their other sources provided.
Jokes aside, I’ve been responsible for a system that processed ~1 billion monetary transactions per day. Even with a fanatical focus on consistency and correctness it was still always off. With the philosophy promoted on the OP the chaos would have been complete… is my gut feeling at least.
E.g. lets say your system is storing billing records for an API. Customers won't complain about being under-charged, so a loss of data for a system like that is fine as long as reducing the durability saves more money than it costs in losts ability to charge your customers. E.g. if rolling up and syncing summaries to disk every 10 seconds were to on average lose you 0.1% of records but cut your server cost for that system by 80%, it's a simple question of whether those server costs are higher or lower than .1% of your revenue (all numbers taken out of thin air).
We make sacrifices in terms of both consistency and durability and all kinds of things all the time. The problem is that we often don't take the time to analyse what kind of inconsistency and loss is acceptable from a business point of view before designing a solution that often tries to aim for a level of consistency or durability that may not be needed.
You can afford to sacrifice all kinds of guarantees as long as you have reliable models for how sacrificing them impacts you and your customers so that you know the sacrifices represent acceptable tradeoffs. The risk you're taking is that you're sacrificing safety margins, and so it's something you should only do if you understand both the domain and your systems failure modes very well.
First off, I'm not sure whether they consider that eventual consistency is not consistency.
Secondly, they quote a long example where business behaviour inherently results in inconsistent data. There is a significant difference between your system being robust enough to let inconsistent data exist, and to give the tools for business users to reconcile it, and a system that is not consistent iself, and lets the users deal with it.
An illustration of this is text editing. If you simply share a text file with multiple people and let everyone edit, it'll be hard to avoid people overriding each other in a chaotic and destructive way. But if you use e.g. CRDTs, you give people the ability to each follow their train of thoughts, and to reconcile their versions later on. That example tends to show how valuable consistency is, and how important it is to build a system that lets users reconcile data if the business logic is not always consistent.
The reason is I've been struggling with the idea that (bi-) temporal, consistent data is great, as it provides a ton of leverage both for users and for auditing and debugging. For some problems it's the cleanest, general solution.
What irks me that the code that validates, consumes, transforms and displays the data lives in its own time model (git).
Philosophically you're not really looking into the past. You're looking at an echo of the past that's displayed in a current form.
And more practically it's simply tough to draw the line between assuring that historical data is still handled and rendered correctly and getting rid of overhead and complexity of a growing and evolving application.
The article talks about eventual consistency, but I really don't have these kinds of problems and when I do, I fully agree with the author as long as it's very clear to the user whether something is consistent and when it will be.
Agreed, the issue of maintaining accurate data (bitemporal or otherwise) in the context of evolving schema and code feels like a real puzzle to solve with commonplace tools like git and SQL.
To the contrary, consistency is extremely important so that they overbook by exactly the right amount, to compensate for the statistically expected no-shows.
And there is no "exactly right amount" that makes it work. They keep options for forcing people off flights if they planned it wrong.
Is also why they don't let gate agents over sell a flight. They keep a stronger consistency on that, for this exact reason. Over selling would fit in what someone else called external consistency.
Exactly same with Amazon, they will not bother checking that the product is actually available because then you will miss the sale when the warehouse service is down or slow, which it would be all the time because it has to be a single place of failure. Its better to refund orders for non existent items than to miss sales.
Consistency requires single source of truth which implies single place of failure. There is tremendous cost to it, it's not done for no reason like you say.
This is not the case at all.
Airlines operate on the principle that a certain percentage of customers are no show. They would like to fly the planes fully booked so they allow some overbooking and they are controlling how much it gets overbooked.
If it was up to CAP there would be 200% overbooking for certain flights and the airline would go bust within a year.
Imagine that you create a multiplayer MMORPG video game with a 'limited' number of virtual gold coins and you set the supply at 1 billion gold coins and you decide to track account balances using 1000 different systems managed by different companies (which constantly credit and debit each other using different mechanisms) instead of just 1 system managed by 1 company... What are the odds that you will still have only 1 billion gold coins in circulation in one year's time? Surely at least 1 of the 1000 systems will go rogue and start inflating the currency supply; it may start sending fake coins to other systems and the other systems won't be able to know that they are fake since they all operate on mutual trust instead of cryptographic proofs, they will unwittingly propagate records of those fake coins to other systems as if they were real. This is essentially how the fiat monetary system works.
The power of money creation is not supposed to be in the hands of private institutions, yet because currency is tracked across so many different systems and there is no universal consistency between them, money creation seems to be essentially guaranteed to occur; especially when you consider that the incentive is to turn a blind eye to errors which benefit large institutions.
https://www.bankofengland.co.uk/explainers/how-is-money-crea...
I also wonder why you think money creation isn't supposed to be in private hands. It needs to be regulated with enforcement, but at the end of the day it is the only reasonably efficient way to couple the productive ability of society with its consumption. The world revolves around promises. Not all will be kept, but the system as a whole is built to handle that and eventually resolve inconsistencies.
That is also why autocratic and corrupt systems are so bad, they put the self correcting parts out of order.
Let's just mix up some numbers and hope for the best.
I mean, what's the worst that could happen? It's not like incorrect data could lead to costly mistakes or missed opportunities. Let's just throw caution to the wind and see where inconsistency takes us. Who needs consistency when you can have chaos, right?
Because a lot of data does.
Network partitioning can completely destroy mutual consistency in the worst case, and this fact has led to a certain amount of restrictiveness, vagueness, and even nervousness in past discussions, of how it may be handled. In some environments it is desirable or necessary to permit users to continue modifying resources such as files when the network is partitioned. A network operating system would be a good example. In such environments mutual inconsistency becomes a fact of life which must be dealt with. [0]
TL;DR - you either store your data in one place, or you store it in multiple places but disallow writes when there's a network split and wait for consensus on all nodes... or you have inconsistency and you have to deal with it.
[0] Detection of Mutual Inconsistency in Distributed Systems, 1983. Great paper, fairly readable, and still relevant. https://pages.cs.wisc.edu/~remzi/Classes/739/Fall2018/Papers...
You can (e.g., using something like Raft) permit writes on the side of the netsplit that can establish quorum (if it exists), which is much less than all nodes.
Delivering consistency in either the CAP or ACID sense requires some amount of constraints on other aspects of the system. Sometimes making this tradeoff it worth it, sometimes it isn't.
The key is to make the value analysis for each problem you face so you can determine the best overall solution to the problem at hand.
So to say consistency is "overrated" is nonsense. It's not always 100% applicable, but this comes at a cost.