Hacker News Reader

Comment by amelius | Hacker News Reader

One of the main problems with breaking up a system into many independent (micro?)systems is that you can't easily combine two or more systems into a logical transaction. Or, in other words, it becomes increasingly more difficult to do things atomically to the complete system.

Is this problem covered?

View on HN

The problem is covered in Chapter 5. There's isn't really a silver bullet, depends a bit on your use case. He outlines a couple of approaches:

* Try Again Later: queue failed parts of the transaction for retry, moving towards eventual consistency

* Abort the Entire Operation: On failure issue a compensating transaction to roll back the previous successes

* Distributed Transaction: have some overall governing transaction manager process that orchestrates consistency (e.g. duplicate the transaction process of your monlithic data store up in the application layer). You could do a two-phase commit where each participant in the distributed transaction tells the transaction manager whether it thinks its local transaction can go ahead. "A single no vote is enough for the transaction manager to send out a rollback to all parties."

For each of these approaches there are issues, obviously, and to his credit he's pretty even-handed about them. He also says:

"If you do encounter state that really, really wants to be kept consistent, do everything you can to avoid splitting it up in the first place. Try really hard. If you really need to go ahead with the split, think about moving from a purely technical view of the process (e.g., a database transaction) and actually create a concrete concept to represent the transaction itself."

He gives an example of a domain model for a retail company of an order-in-process: instead of attempting to capture a user's order as a single database transaction, model the concept of an order flow of through fulfillment, shipping, marketing, accounting, etc.

A concrete example we've faced. A certain operation requires writing data to N flaky services. You successfully write to N-1 of them, but the Nth fails. Now what do you do?

If these N things were just database writes to the same DB, transactions would save you, as you could just rollback. Without that, the answer has to be handled in code -- do you reverse the previous changes (if possible) by sending delete events, or leave the system in some sort of half-baked state and rectify things later via some other process? (I'm interested in hearing of other options...)

On my phone so boo cite. Look up Amazon apologetic computing. They also have a presentation describing their general distributed approach. The main thing is that you have a bunch of components that you expect to fail. If they don't get confirmation, they send another message to the next component. Then, there's a stateless, unambiguous way to determine which message "wins" in case of conflict. For instance, an order submits two shipping requests for the same order. They use, for instance, the highest message ID (a GUID) to avoid duplicate processing.

It's all very neat, though I'm doing a bad job explaining. Search for Amazon architectural presentations and hopefully you can find it. It may have been on James Hamilton's blog, maybe.

You are basically describing the consensus problem [1] and one of the most popular algorithms used to solve is the paxos algorithm [2]

[1] - http://en.wikipedia.org/wiki/Consensus_%28computer_science%2... [2]- http://en.wikipedia.org/wiki/Paxos_(computer_science)

As far as I understand microservices, these kind of issues must be solved by your architecture. Either the service is atomical, or the orchestration of your services must be in "flow", where no race conditions should occur.

It sounds like this is a difficult path to take. Especially considering the fact that, usually, system requirements may change over time.

It absolutely is. It's for exactly these reasons why only highly organised and qualified teams tend to have substantial microservice deployments today.

It's also why I think[1] we need to start building a framework[2] to provide standardised solutions (or at least patterns) to some of these problems.

[1]: https://vimeo.com/118895501 [2]: http://wym.io/

Reply on news.ycombinator.com