Hacker News Reader

Comment by ecopoesis | Hacker News Reader

Almost every distributed system (including "simple" client-server systems) is eventually consistent. And all systems are distributed.

It's great that your DB is ACID and anyone who queries it gets the latest greatest but in reality you also have out of date caches, ORM models that haven't been persisted, apps where users modifying data that hasn't been pushed back to the server and a million other examples.

I'm sure it's possible to create a consistent system but I'm also sure it's not practical. No one does it.

Instead of constantly fighting eventual consistency just learn to embrace it and its shortcomings. Design systems and write code that are resilient to splits in HEAD and provide easy methods to merge back to a single truth.

View on HN

There is a huge difference between having an ACID store of Truth surrounded by eventual consistency, vs making even your store of Truth eventually consistent. You're basically doubling or tripling your work for any given constraint because you have to both monitor after-the-fact violations and build in a way to resolve those violations.

This is on top of regular "nope, can't do that" code that you would write in both systems.

It's just the opposite, in my experience. If you have an ACID database that's supposed to represent the current state of the world, you have to handle both transaction rollbacks and logical inconsistencies. If you have a streaming system where you record an event log and generate the current state of the world from that, you already have the logic to recover from inconsistencies and can reuse it.

> I'm sure it's possible to create a consistent system but I'm also sure it's not practical. No one does it.

Billions of dollars flow through fully consistent systems every day. The basic IT concept for smaller hedge funds is "buy the biggest MSSQL machine available on the planet and move on." The big ones have custom frameworks that resemble Frank's arguments here, though the abstractions are different.

> The basic IT concept for smaller hedge funds is "buy the biggest MSSQL machine available on the planet and move on."

And the result is exactly what grandparent was complaining about: sure your database server is full ACID, but a trader is looking at numbers on their screen that are out of date (and pressing the trade button on that basis), and that's what actually matters.

> No one does it.

Oh, some people do. I used this EXACT phrase when I came in to fix an analytics system at a healthcare company that was plagued with analytics problems. They had 5 senior engineers, fulltime, working on this system for years. It had persistent problems and could not be modified in any meaningful way. Upstream systems sent data through multiple SQS topics (duplicate and out of order data) fed into lambda, fed into a giant cache-db which tried to catch dupes and order data, fed into files, processed in batch. It was a horror show in complexity and costing (despite the near-free lambdas). A distributed set of large data streams we feeding into a singular database which was processed, multiple times and put back in the same database. What's billions of inserts into an amazon postgres db, per hour? The company cloud infrastructure gave 0 other tools to work with. I shored up the batch processing (which had all kinds of try catch everywhere, despite a fixed schema) and went on to another company. Medical company software is always a ball of fail.

> I shored up the batch processing (which had all kinds of try catch everywhere, despite a fixed schema) and went on to another company

I was hoping for a happier ending there. What could have potentially fixed their situation?

My company is an automated insurance broking service which issues about 4% of all UK motor insurance policies (by number, not value).

We have consistency across our distributed system (~75 services currently) for all the fundamentals of our business. It is not difficult to do at all.

Reply on news.ycombinator.com