Hacker News Reader

Comment by winstonprivacy | Hacker News Reader

That is precisely what I'm doing now and it's the best solution I've found so far. However it is brittle and it takes quite a long time to iron out the edge cases where things can happen that you didn't anticipate.

I have one system that has been more or less stable for two months and once a week, it fails inexplicably (dumping no logs either). Whenever this happens, I find myself writing a ton of debug code hoping to catch it.

View on HN

I worked on a highly complex, high-volume automated procurement sourcing system many years ago. Each Purchase Requisition (PR) -> RFQ -> Bid Response -> Purchase Order could have many-to-many relationships, splitting & bundling at each line item (even the arrows above don't give the right idea, as things could go to re-bid, long-term contract with task orders, etc.).

We tried for nearly two years to get the status/state of each PR line item to update in real-time as events occurred during follow-on steps, but it was extremely brittle.

We finally added an intermediary queue and centralized all the logic in batch. Anytime something happened that might trigger a state (not verified), we tossed it in the queue. Then, we processed the queue every five minutes through a rules engine that was a classic factory pattern (a favorite anti-pattern). Every time the PR # ran through the rules engine, it would have a deterministic result. Sometimes it would pass through 10-20 times with no state change, but at least it was predictable.

We thought it would be too inefficient, potentially processing the same PRs 100s of times without any actual change to the state. However, the servers handled it fine, and it was real-time "enough" the users didn't notice a lag. We went from fixing issues several times a week to no changes for the remaining two years I was on the project.

Not sure about the specifics but yes, in cases like the one I described in my oriignal post, the "header" state will not change immediately but some kind of service thread would periodically revisit the active headers (i.e. anything that had not reached the final stable state) and check all the items in order to update the header state.

That worked fine for us. Also, I have recently (< six months ago) used a SM to build a module in the application I am working on and I am quite happy how it allowed us to quickly adapt to the corner cases that inevitably started cropping up after deploying the module.

I am fairly convinced that a different approach would have make been much more difficult to adapt, and that testing for subtle interactions of different states would have been much more complicated.

I agree on the fact that you may have edge cases cropping up in production... still, in my experience this approach makes adjusting (either by introducing new States, or by altering the logic that guides a State Transition) much easier because you can reason about the problem in terms of a specific item and a specific State Transition at the time, even if this happens to 10k records at the same time, and there are another 10M records of different type and in different states.

I used a SM no more than six months ago to process vouchers and voucher redemption from a third party service and while we had no less than 10 changes after deploying in prod... each one was easy to analyze and the actual fix was like 2-3 lines of code.

Reply on news.ycombinator.com