Event Sourcing (2005)
martinfowler.com
martinfowler.com
I've used the event sourcing pattern with great success in the context of an exceptionally complex web application for my day job. The front-end would create an 'event' for every action the user took. This allowed us to seamlessly implement client-side undo/redo functionality with surprisingly little effort, a market leading feature our users really appreciated.
One key insight we had was the necessity of capturing the state change as a difference, as Martin explains. This turned out to be tremendously powerful. We were able to build a variety of other features on top including detailed analytics of how users were using our app, and session replay for training and debugging, among other things.
How CRDTs work is different in different implementations. Some CRDTs use a timestamp to order all changes, thus making it essentially identical to an event sourced list.
Timestamp based ordering may not actually be the most ideal, especially for low trust environments where the timestamp can be spoofed.
Some apps might not last-write wins. In this case, timestamp largely becomes meaningless because there has to be some other way to resolve conflicts due to multiple clients making changes to the same piece of data. Some apps want one clear winner, other apps might want the two inputs to be merged, others (like git) explicitly ask the developer to resolve the conflict.
In this way CRDTs can diverge significantly from an ordered list of changes.
> ... all the capabilities of reversing events can be done instead by reverting to a past snapshot and replaying the event stream. As a result reversal is never absolutely needed for functionality. However it may make a big difference to efficiency...
When I implemented undo/redo, I found that replaying from snapshots was fast enough. I wasn't working on a database-backed app though.
Implementing reversal is extra work and bug surface, so it should be subject to cost/benefit analysis.
I usually pair that with a "check" function which just verifies that all the invariants I expected hold true. Eg, check the user has a non-negative balance.
It sounds complex, but you get massive bang for buck from code like this. 100 lines of fuzzing code can happily find a sea of obscure bugs.
I like this example as a demonstration from FsCheck using F# (for the test program) and C# (for the program under test): https://fscheck.github.io/FsCheck//StatefulTesting.html
It's essentially as you described.
https://github.com/josephg/skiplistrs/blob/master/tests/test...
In a loop it simply randomly decides whether to insert, delete or replace some text, and after any action checks that the state is valid (In this case via a call to check2()).
You can go way deeper with this sort of thing if you want, with more complex models, random item generation and simplifiers to help pare down problems to simple test cases. And you can have more complex state - for example, where you also track interaction between multiple items. For example, if your state is a few users, and you also track one user paying another user and verify the total balance across all users is unchanged.
But you don't need to go deep for fuzz testing to be worthwhile. Even something as simple as this little loop is a remarkably effective bug finder.
increase balance by 10 (it is currently 100)
balance is now 110
increase balance by 10 (it is currently 100) <- duplicate event!
event has already been applied
balance is now 110
Combine this with versioning, where you assert an event stream version on write, and now you're able to affect change on a known state that cannot have changed (and if it did change, your command would fail and cause no persisted changes server-side).
Not necessarily. Depends on how you shard the aggregate/streams. Since aggregates _by definition_ maintain their invariants, you can have literally a shard for every aggregate. Overkill, but it illustrates the point. Since there's no state shared between aggregates, you can process each aggregate/stream on its own instance.
> this instance checks all command ids if they appeared before (so keeps track of all ids in an efficient way)
Yep. If you trust the client and don't mind losing old commands, this isn't too hard - use a monotonically increasing number (e.g. ms since the epoch) as part of the commandId. Any command with an id less than the commandId in the aggregate will be rejected as "out of date". Note that this serves as an optimistic concurrency check.
If you don't trust the client (or you don't want to lose old commands issued on a possibly out of date view) and just use GUIDs as commandIds, then yes you'll need to keep track of some ids - but not necessarily all. Do you really need commandIds past 1000? (Depends on your domain.) You'll need to "roll up" the events into some summary view anyway to check/maintain business invariants, so keeping, say, the last 100 commandIds as part of that summary view is simple.
Now I only wish there would be a good alternatives for when you want to have multiple instances per aggregate (even if you may have to give up certain guarantees).
If you're trying to do multimaster db storage, like a phone app syncing with some cloud server, where an aggregate resides in two separate locations, this is where event sourcing shines. It's just like git, right, and so you can do all the stuff that git can do - merge event streams, rebase events... it's actually the reason why my pet project switched from an RDBMS to event sourcing. Related video: https://skillsmatter.com/skillscasts/1980-cqrs-not-just-for-...
What literature, or framework or ecosystem holds this to be a hard-and-fast requirement of CQRS?
Personally I've solved this problem similar to Git, modelling CQRS on top of Merkle trees so your system at least knows if your incoming edit (command|commit) is being applied on HEAD (no changes to the model since the command started) or to HEAD+n (needing to re-run some command validations, and "rebase" the incoming command).
- idempotency is ONLY a hard-requirement if the events are transmitted via lossy channels such as network. ID is one out of several ways to ensure idempotency. If you're implementing CQRS for your distributed system then it's needed - otherwise, if the events aren't lossy and can be processed atomically, you don't need to ensure idempotency. think implementing CQRS for your user interface where events are just transmitted between different objects in memory rather than via network
Well, that is exactly what OP meant - in the context of a distributed system it is not "at some point" it is just a given.
Balance is 100
Operation #1 says add 10
Balance is 110
Operation #1 says add 10
Balance is still 110
Basically the source of the debate - to do "proper" event sourcing, do you need to rerun the computations each time you roll back/forward, or is it enough to simply restore state?
Next year on the Q1 I'll be able to push for a full demo again. Hope we get traction this time. It's great tech, and a pleasure to use.
Apply seems to be the common notion of event sourcing: There is a function apply that takes a state an event and yields a new state. Then, starting in an init state and iteratively applying the entire event history, boom, latest state restored.
Dedup has a lot of charm though: Run and rerun your code, if that step of your code is executed for the first time (no corresponding event in the event history) execute the step and store its result as an event in the history, however, if that step of your code is executed for the second, third time (there is a corresponding event in the event history) do not execute the step and return its result from the event in the history. The Haskell Workflow Package (https://hackage.haskell.org/package/Workflow) is a good example
Temporal follows the second approach, so "proper" Event Sourcing? You be the judge :)
This of course would be incredibly reliant on the code not being able to access anything except the state, and have no side effects - so if you want to say "if X, fetch Y" you need to have something looking at derived state from the "if X" part and putting the results of "fetch Y" into the event stream, thus causing the state relevant to the code to change, and invalidating the need for something to "fetch Y" a second time? There's something incredibly clean and enticing about this, but also incredibly scary to implement correctly at scale!
My most thorough use of event sourcing was in a real-time art auction system, which combined in-person and online bidding. An operator had to input the actions in the room into the system and bid on behalf of online bidders. Event sourcing allowed us to model what actually happened for each auction lot. It also made it possible to model undos and cancellations in really slick ways.
Command-query response segregation (CQRS) is also a powerful concept, and it combines elegantly with event sourcing. I also find it to be a handy way of thinking of raw and derived data, rather than trying to perfect a normalized data model to serve all purposes.
I built https://batch.sh specifically to address this - not having to reinvent the wheel for storage, search and replay.
For some cases, storing your events in a pg database is probably good enough - but if you're planning on storing billions of complex records AND fetching a particular group of them every now and then - it'll get rough and you need a more sophisticated storage system.
What storage mechanism did you use? And how did you choose which events to replay?
All command processing and state queries required us to read and process the whole event log. But this was fairly cheap, because we're talking maybe a couple dozen events per lot. To optimize, we might have considered serializing core aspects of the derived state, to use as a snapshot. But this wasn't necessary.
Batch looks pretty cool! I'll keep that in mind next time I'm considering reinventing the wheel :)
Event sourcing is not easy but the benefits are huge - not only from a software architecture perspective but from a systems perspective - you gain a whole lot of additional platform reliability.
There are roughly 3 pillars to event sourcing that have to be built (most of the time) from scratch - storing events (forever), reading stored events, replaying stored events.
Those are the 3 things I've built at several companies and it is always a huge barrier to someone taking up the pattern. Of course there are more things under the hood, but those are foundational pieces that will make the whole experience much better.
Would love to chat with folks who want to nerd out over this stuff :-)
From what I have seen however, industry is much more technologically diverse than people imagine.
Event sourcing is an amazing tool. The failing projects are usually the ones that want to use it without acknowledging that it offers different tradeoffs from more traditional data modeling/storing solutions, and not adapting to meet different ups and downs.
There are also many projects that want to use event sourcing, but actually end up picking up event sourcing, reactive programming and a host of frameworks or even languages to do the second. That's a lot of technical baggage to learn at once, and people making the switch in a production context with tight deadlines and without deep pockets to hire experienced people usually don't fare so well.
Basically it follows the git model - distributed systems are each their own sources of truth, and allows for (relatively) simple syncing of state. When two distributed systems conflict, well... you need to write some kind of merge algorithm that may or may not ask for user input. But at least the accidental complexity is significantly reduced.
I ended up using Redis Streams [0], which was good enough for my small-scale need. We already had Redis in our stack too, so it was a very simple integration.
It works pretty well this way, not everything needs to be streamed.
I'd be happy to walk you through the platform - there's no lock-in, since we don't require the use of any SDK's, just run an event-relaying container which will pipe all of your events to our stores.
One big piece is that our platform is message bus agnostic meaning, we are able to hook into any message bus, be it kafka, rabbit, nats, etc.; same goes for replays - we can replay into any destination.
Check it out: https://batch.sh
The relayer is open source - https://github.com/batchcorp/plumber - if anything, the relayer can also be used for working with message busses which could improve your dev workflow for reading/writing messages, etc.
The whole thing was super complex to understand, the event flow that could be triggered made it hard to understand what was going on and migrations where very difficult to do correctly. Notifying frontend with "eventual consistency" was pretty terrible to.
Event sourcing is a very powerful concept, but it’s very easy to use inappropriately. If you can’t map your business/domain problem very easily into the paradigm, it is worth pausing and reconsidering. Don’t get stuck maintaining a system you never should have built in the first place.
You can also replay states using a special “rewind” mode, a core advantage of event sourcing.
I came across the Update Monad [1] while researching how to use Event Sourcing in functional languages: it basically has the same guarantees of only changing the state through recorded events. Beware though, as it can be quite slow if implemented naively.
https://hn.algolia.com/?q=https%3A%2F%2Fmartinfowler.com%2Fe...
I am still struggling with the grain on which I would want to source events. The engineer in me says its safest to go low-level and source key-value, so it doesnt matter if the biz model changes. But, this would reduce fidelity on the other side in terms of scanning for specific events of importance (e.g. SetKeyValue vs SetCustomerEmail).
Basically, I vote for `SetCustomerEmail`.
This is clearly the key to the whole picture.
Reading through this and other comments this morning, and I am starting to develop some concrete notions.
Seems like I simply need to build events for each unique type of fact that could exist in the domain. The verbosity of this seems extreme, but also makes a ton of sense.
The thought in my mind right now - If you combine event sourcing with a model in 6NF, the mapping should be trivial. Every fact table has its own series of events pertaining to it (Create/Update/Delete).
This also seems elegant - If your core data representation is inherently linear, do you care that you have this degree of normalization going on? You have to reconstruct some reasonable in-memory representation from the log anyways, so your live in-memory instance could be ~3NF while it ultimately transacts with a 6NF event log on disk. This also plays nicely with backwards compatibility and data migrations.
For small changes like adding a field, we make the property on the underlying event optional, but require it in the event dispatch function. That allows old events to be missing the field, but type-checking still requires it in the codebase going forward. It does require a special case to handle the missing field, however.
For larger changes to events, we do migrations. We store events in Postgres, so we just use TypeORM migrations. I haven’t had to do one of these myself, but I’ve heard secondhand that it’s not too bad. We don’t have very many of these, regardless.
I inherited an application with 24M+ events and I have no idea how to go about it.
It's not really different than handling regular backups. You don't wipe them, just guard them well until they expire.
Re 24M+ records: create a batch runner that goes through "jobs" to perform stripping/cleaning tasks. To store state (and to organize cleaners), use a distributed store such as etcd - that way you can bookmark where you were at in the cleaning process.