Comment by eric_b | Hacker News Reader

Comment by eric_b | Hacker News Reader

Parent Full thread

One of the projects did use Kafka as the "event log". There were stability issues with the version of Zookeeper that was used. From a writer/reader perspective Kafka was sufficiently performant when it was up. (The Zookeeper issue was eventually fixed as I recall, but by then the damage was done in terms of political capital spent and lost.)

The big issues didn't really have that much to do with the persistent store of events. The bigger issue was the fact that as new features get added to your application, your event payloads change. Well, in order for projection (particularly re-projection in the case of an issue) to work, your code needs to know how to read and process all versions of every event. Of course, there are techniques like snapshotting to give you point in time "good states" so you can eventually deprecate some of these events, but thinking about it is challenging.

Additionally, most folks argue that the event store should be immutable. This is great until some kind of bad event gets in there. Now your code needs to know how read, and discard, this bad event forever (or until a snapshotted point in time).

Finally, projection is not the panacea the evangelists will have you believe. Inevitably there will syncing issues between the event store and the projected database/elasticsearch cluster/mongo instance whatever. And what do you do then? Re-project! But that is not easy :)

We use a flavor of event sourcing in production (Elixir+Postgres) and I have found many "rules" can be broken while still maintaining the core benefits of CQRS/ES

- We use Postgres as the event store and all of our other projections are stored in the same database.

- Our app is not distributed. We use a single database.

- Our event store is not immutable. Instead we will run migrations to rewrite the events, delete events, etc. You either have to either deal with the complexity of maintaining two versions of code or the complexity of migrating. I've found the later is a fixed cost (do it once and move on) vs. a variable cost (continue to deal with two versions of code).

- Our commands aren't async. They are executed inline.

- We don't do any snapshotting.

Granted, we don't have a lot of users on our app (~50 active users) but overall this has been a positive experience.

We use the strategy of versioned events. Event data is still immutable, but older events are upgraded to the latest structure at read time. This works reasonably well but it is not ideal.

Storing data as immutable events implies that all data ever generated by your application becomes available to future versions of your application. Writing an application that can handle all the forms of your data across time is obviously more complex but it's a necessary consideration if you decide to go with event sourcing. Unfortunately, you cannot have all the benefits of available and useful unaltered historical data without also putting in the engineering effort to support it.

For migration of immutable events, there's a good research paper[1] that outlines five strategies available: multiple versions; upcasting; lazy transformation; in-place transformation; copy and transformation. The last approach even allows you to rewrite events into an entirely new store.

[1] The Dark Side of Event Sourcing: Managing Data Conversion http://files.movereem.nl/2017saner-eventsourcing.pdf

To address the concern with changing "immutable" events. I've just built a migrator [1] tool for my PostgreSQL-based event store. It implements the copy & transform migration strategy. The source event store database is untouched. Transformed events are written out to a separate database.

It uses Elixir streams to provide composable transforms to rewrite, aggregate, remove, and alter serialization format of events.

[1] https://github.com/slashdotdash/eventstore-migrator

Reply on news.ycombinator.com