163 karma · joined January 13, 2021
TLDR: AWS didn't want to pay for licenses and rolled out their own thing.
As Op said, not needing to rewrite applications or using the muscle memory from using Mongo is beneficial. I'm not planning to be strict and support only MongoDB API; I will extend it when needed (e.g. to support raw SQL or JSON Path). But I plan to keep shim with compliant API for the above reasons.
MongoDB API has its quirks but is also pretty powerful and widely used.
Regarding IDs, you can use any UUID-compliant format.
Of course, rebuilding a state from events each time you're processing command may sound dubious. Still, event stores (even those with relational DBs as backing storage) are optimised to read events quickly. Typically, reading event 100 events are not an issue for them. Also, for a typical LoB application. The temptation to use Snapshots is strong, but it should be treated as an optimisation when there is huge performance requirements. They may get out of sync with events. They may be stale, etc. I wrote about that in: https://www.eventstore.com/blog/snapshots-in-event-sourcing. The critical part is to keep streams short. That's again a difference between Event Streaming and Event Sourcing. In Kafka, you may not care how long your topic is, as it's just a pipe. Event stores are databases, so the more events you have, the worse. It impacts not only performance but also makes schema versioning harder (more on that: https://event-driven.io/en/how_to_do_event_versioning/).
It's easy to fall into the trap, as using a streaming solution is tempting. They promise a lot, but eventually, you may end up having issues as you described.
Still, that can be said on any technology. Event Sourcing has some dark parts. Also, it's essential to highlight them, but it's dangerous to present other tools and patterns issues related to ES.
Event Sourcing by itself doesn't directly relate to eventual consistency, type of storage, messaging, etc. Those are implementation details and tradeoffs we're chosing. Each storage solution has it's patterns and anti-patterns. Relational databases have normalisation. Document databases are denormalised. Key-value stores have strategies for key definition. Event stores also have their specifics. The most important is (as I mentioned) to take into the account the temporal aspect of streams, so keeping them short. I agree that there is a huge gap missing in knowledge sharing. I'm working on the article recently about that aspect. Here's a draft: https://github.com/EventStore/blog-articles/blob/closing-the....
I think that you highlighted well what can happen if you use the Event Streaming solution as a basis for the Event Sourcing solution. But unfortunately, the title and description state that it's about Event Sourcing, which is unfortunate, as it's highly misleading for people who are not aware of what Event Sourcing is and may suggest that it's much more complicated than it's in reality.
See also my other comments below in the thread, where I commented also in more details on other parts.
Event stores can integrate with the streaming platforms, to publish events and move them forward. Most of them have even built-in implementation of the Outbox Pattern that allows to publish events. They're usually called subscriptions. They can be used to build read models or triggers other flows, or route events to streaming platforms.
The essential point is that for read models, you don't expect to have the same guarantees as for write models. It's fine to be facing idempotency, even out of order, as read models shouldn't be used (in general) for business logic validation. In Event Streaming, or models are stale. In Event Sourcing, write model is not stale, you can be sure that you're making a decision on the current state. Thus, most of the points mentioned in the article, are just from struggles to use event streaming solution as tools for storing events. Which is not the issue of Event Sourcing or event stores per se, but using the wrong tool for the job.
`the raw event stream subscription setup kills the ability to locally reason about the boundaries of a service.`
`you have to talk to the people who will be consuming the events you produce to ensure that the events include enough data for the consuming system to make a decision`
`wire a fleet of services together via an event stream`
etc.
Plus the drawing that's at the beginning.
This clearly states that the author doesn't understand what's Event Sourcing and tells about Event Streaming. Event Streaming means:
1. Producer pushes/produces/publishes an event to the queue. In this sense, an event stream is a pipe where you put an event at the end and read it on the other side.
2. Then, you have a set of subscribers that can react to those events. The reaction may be triggering the next step of the workflow. It might also be updating the read model state. That happens with eventual consistency. You also cannot guarantee atomicity or optimistic concurrency, as tools like Kafka doesn't support it. As they were not originally built for. They let you effectively publish and subscribe to the messages.
3. Because of that, if you want to have the write model materialised from an event (or even "populated"). Then you don't have any guarantee if it's stale or not. You need to fight with out of order messages and idempotency. Thus all of those issues that are described in the article.
I'm sorry to say, but the author used the wrong tool for the wrong job. If this article be titled "Event Sourcing is hard if you're confusing it with Event Streaming" then it'd be an excellent article. But, in its current shape, it's just misleading and making the wrong point, repeating just common misunderstandings.
You can read more in: