Hacker News Reader

> the receiver can easily check if a specific message was already received

It seems like you're conflating exactly-once delivery with exactly-once processing. Your proposed solution is only necessary _because_ we can't have exactly-once delivery as TFA states.

I don't agree with that contrived definition of delivery. As a developer, what I care about is delivery of the message to my function/handler. E.g. so that I can store it in my local database/store on the receiver side exactly once and ultimately show it to the user exactly once. Definitely possible.

>> we remove the need for idempotency through deduplication.

> If the client generates and assigns a UUID to each message it sends, the receiver can easily check if a specific message was already received before by comparing it against previously received UUIDs and can discard duplicates.

You quoted the part of the article that suggests exactly your solution: deduplication.

Yes but the author suggests that it's difficult to perform or requires constructing messages in a complicated way which is not the case. Generating and attaching a UUID to a message is very simple, so is checking whether or not a UUID has already been encountered on the receiver side.

The author's definition of 'message delivery' is incorrect and so is the title which is clickbait.

Perhaps adding a UUID is too much overhead? If you have a very high throughput system with messages of less than 128 bits, then the addition of a UUID to every message would effectively cut throughput in half.

I do not have such a use case in mind, but the description above does not sound terribly unrealistic to me.

Also in order to track UUIDs that have been received, you would either need an infinite amount of memory, or have the system bounded in terms of how many messages out of order a message be delivered and still be processed correctly.

You can use a database. Practically, most systems will keep a record of each message on disk in any case (at least they will have some logs) so it's not that outrageous. Safe to assume that the receiver can discard non-authenticated messages to avoid spam.

It depends entirely on volume of messages vs available memory (ram or disk) how outrageous it is

You’re completely missing the point. Many multi master databases exist. Network partitions happen. Bugs in multi master databases during network partitions have happened and can happen. This is like distributed systems 101.

So given those facts, your design of using UUIDs is absolutely not full proof if the database where you stored your UUID is not in a consistent state.

Here’s some more useful info for your perusal (https://ucare.cs.uchicago.edu/pdf/socc14-cbs.pdf)

“ Data consistency means that all nodes or replicas agree on the same value of a data (or eventually agree in the context of eventual consistency). In reality, there are several cases (5%) where data consistency is violated and users get stale data or the system’s behavior becomes erratic. The root causes mainly come from logic bugs in operational protocols (43%), data races (29%) and failure handling problems (10%).1 Be- low we expand these problems.”

Your database does not have to be networked, it can be on-disk KV store like RocksDB, which is what things like Flink state backend use.

If all of your agents are seeing the UUID for the first time, they cannot deduplicate amongst themselves without quorum or shared state which has race conditions itself. It’s non trivial.

There's no need to. Different agents will have different copies of the same message, that's a given. If the sender ensures that a UUID is attached to a single specific message and never re-used (though they may retry to send the exact same message multiple times), then there is no risk that multiple receivers would have different versions of a message and therefore consensus is guaranteed without even having to talk to each other.

If you assume that the sender cannot be trusted to not reuse a single UUID for multiple messages (e.g. to trick the receiver), you can still work around that limitation by computing and comparing message hashes (e.g. sha256) on the receiver side... Don't even need UUIDs. Every time you receive a message, you can hash it and check if you're received a message with this hash before (storing the hash of each message as it is received). You can use the hash in place of a UUID though in this case you probably need to add some index to each message to ensure that each hash is unique over time (since a message with the exact same payload will be counted as the same message, even if broadcast a long time apart).

exactly once delivery is not satisfied if the same message is delivered to the recipient multiple times

It depends on your definition of 'recipient' and 'delivery'. If you assume that the receiver is a specific database instance (or shard) or a specific data store (on the receiver side), then it's totally possible to have exactly once delivery in the sense that each message sent would end up being inserted into the database/data store exactly once without duplicates.

but you can't make those assumptions, in general

Yeah, non trivial is correct.

A common Kafka approach is to partition by key, so that a given UUID will only be placed on one partition, and we're guaranteed that any further messages with that key will also be placed on that partition, so handled by the same consumer.

Then create a change-log topic that's co-partitioned by key with the input topic. And then funk around with partition assignment strategies so if consumer X is assigned partition 10 of the input topic, then it's also assigned partition 10 of the change-log topic.

Then add RocksDB as a fast state store on a persistent volume claim, as restoring state fresh from your change-log topic turns out to take about 7 minutes.

And then realise you've just reimplemented bits of Kafka Streams poorly.

But not impossible.

Depending on your ability to share state between these agents it could very well be impossible. It depends on other factors.