Hacker News Reader

Kafaka you get topics and partitions. Topics is a grouping of data, normally having a common schema. Partitions allow parallelism spreading the data across multiple brokers.

Data on a topic can be partitioned by a key so you can partition by user id resulting in all user events going to the same partition however there’s no functionality to delete a partition by key.

If each user has their own topic that you can delete you would end up with a lot of topics and Kafka isn’t great with that.

You also have the issue if event sourcing or doing DDD your events are based around domains/aggregates with clear context bounds. The domain is very likely not a user but may involve a user. Its a tricky modeling issue.

Event sourcing is about immutable events. You cannot change an event in the past as its already happened, accounts don’t use rubbers. The ability to go back and change past events or delete them breaks the laws and guarantees of an immutable event system.

It’s a difficult problem. I like event sourcing due to the properties it gives but it may not be suitable for some things due to GDPR.

One thing I have heard talked about is encrypting personal identifiable information in an event with a user specific key. When the user requests deletion you delete the key. The events don’t change they are still there however anything which identifies a user is unreadable.

Indeed - we designed ours in a similar way as you describe and whilst GDPR causes a small bit of work for us it is not like we face a total design flaw type issue that it appears people are experiencing with Kafka.

We are looking at adding a Nuke command + Nuked event that will serve to destroy all data in the aggregate root snapshot and leave behind (effectively) a tombstone snapshot. The command handler will then purge all prior events leaving behind just a Nuked event. This means the event stream doesn't just disappear entirely, it can still be synchronised out to dependent systems, and indeed those systems will learn of the nukage just like any other event.

What (if any) is your favourite book on event sourcing?

I've seen tech meetup lectures (one particulary fascinating one was a guy who worked for the NHS) but I'd like to understand it better than I do.

To me the biggest thing that made event sourcing click was when I was working on an internal app that needed full auditing. With event sourcing the audit log is the authoritative data source.

Learning that there was a name for this and that others had worked it out better than I had was very handy. I didn't find a need for a book because I found reading Martin Fowler's article[0] on it to give me a good enough understanding.

I've found that it's a less well-defined pattern than some implementations make it seem. There are many trade-offs to be made. At the one extreme you can treat events as physical events that can be easily understood. Let's say we have a user that wants to add a phone number. Well, we could have an AddPhoneNumber event. But this leads to a RemovePhoneNumber event as well.

But you can also take the other extreme and say that each event is a set of collected information -- perhaps a different element of the root of a json object -- and that you can just diff to see what changed and only need to look at the latest such event.

I've found there's a middle ground that doesn't require too much thinking: related data goes together if it's not going to change very often. So I will generally put all contact information in together as an UpdateContacts event. This way we don't need to implement all the list operations for phone numbers and can skip looking for such events after the latest has been found. You also still have something simple enough to work out what changed and that is unlikely to fill your storage.

If you are implementing event sourcing, I would like to point out one thing that I didn't see written anywhere: set a reasonable upper limit on per-user storage. Because you will find somebody who, even not maliciously, just keeps changing things back and forth and it's better to stop them adding events than to stop all users by running out of disk space.

[0] https://martinfowler.com/eaaDev/EventSourcing.html

Thank you for the thoughtful response.

I was hoping for a good book of the "here be dragons" kind.

Often books on an architecture gloss over the rough bits and then you find them yourself when you already committed to it.

I've become incredibly conservative when it comes to application architecture the older I get.

If you're going to dive into CQRS/ES, I'd recommend:

* Enterprise Integration Patterns (basically an entire book about messaging architectures) [1] * Vaughn Vernon's books and online writing [2], * Domain Driven Design by Eric Evans [3], * and most of what Greg Young, Udi Dahan, and that constellation of folks has done online (lots of talks and blog articles.)

Depending on your platform of choice, there may be others worth reading. For my 2¢, the dragons are mostly in the design phase, not the implementation phase. The mechanics of ES are pretty straightforward—there are a few things to look out for, like detection of dropped messages, but they're primarily the risks you see with any distributed system, and you have a collection of tradeoffs to weigh against each other.

In design, however, your boundaries become very important, because you have to live with them for a long time and evolving them takes planning. If you create highly coupled bounded contexts, you're in for a lot of pain over the years you maintain a system. However, if you do a pretty good job with them, there's a lot of benefits completely aside from ES.

[1] https://www.amazon.com/Enterprise-Integration-Patterns-Desig...

[2] https://vaughnvernon.co

[3] https://www.amazon.com/Domain-Driven-Design-Tackling-Complex...

By default Kafka has no mechanism to delete records from the event store, apart from a TTL.

You can enable topic compaction which can be used to delete records by overwriting them, but compacted topics just don't make sense for a lot of Kafka use cases.

So the only option left is the one suggested in this article: encrypt everything with individual keys and delete the key (rather than the data) to comply with GPDR

Your idea is right, but it really depends on the event-store implementation. For example, Kafka has a limited amount of topics (10K if I remember well), so unless you have a very limited audience, I don't think having one topic per user is the solution over there.

Right. This makes sense as to why it is a problem for Kafka users. I must admit I was in a bit of a hurry when I looked into Kafka to see if it would be a better fit, but it seemed to be more about the scaling than I cared for.