It works alright (like you I hated it all before, coming from more amateurish implementations). It's slow to change (adding a new event type can take years before it works everywhere), failures can happen - for instance if the message parsing library has a crushing bug and an extendable attribute zone has a poison pill that never appeared before - well nothing you can do but manually editing the event source.
What it brings us, I suppose, is that every micro service is single threaded, all events are well-recorded , we use multicast to transfer them from one sequencer to all consumers so we just need good routers and TCP-level message building - it's very barebone to keep it fast, extensibility for us is mainly on adding more services around a core stream that we don't really need to change all that much (we do a lot of regulatory validations, data analysis, the odd scale out for a round-robin compatible process - not all of them are, some need to see all events, for instance for cumulative exposure calculation - client stock exposure on several markets for risk-based decisions).
It also avoid latencies like in state-request based systems, since each service will build its own state machine. We make a lot of money on this system, so we can hire hundreds of people around the world to maintain it.
At this point I dont see how to do it better (5ms round trip to the client if low validation, 100ms if crossing seawater to an exchange with short selling validations) without events, but I know very well that doing it for simpler flows that are not latency-constrained will probably result in heavy cost and low gain. I would never recommend it for people who aren't already struggling with a fully-fledged business implementation that makes money they want to accelerate.
The problems we face are:
- it's slow to evolve the very core of the system
- we need perfect ordering, there will always be time wasted at the sequencer to transform unordered "commands" for the various services into perfectly ordered events
- testing and debugging is an art that takes time to acquire: I can now, but the task seemed daunting when I started - how to spin up the minimal surface of services to make a valuable replay, how to make static configuration reproduce production's behavior exactly so that all services behave the way you want to reproduce if you're investigating a sequence-based issue (rather than a function-based issue)
- it takes up to a year for a new Java dev to get productive on such an exotic mindset, but it's also because we do no intraday malloc since we cannot afford a GC in the middle of a client order
- management cannot understand why they can't cut cost using the cloud, virtual machines, vendor databases etc. Even in a company that makes billions over 20 years on this system, we still can't explain it in a way that sounds valuable vs its cost. Because its cost probably is extremely high, and can't be outsourced by hit and run consulting managers before being brought in-house again. So we're not like the most popular dev, we're the slow and expensive ones :(