Hacker News Reader

Comment by jeffbee | Hacker News Reader

Head of line blocking as a service!

Seriously, I can't shake the feeling that most people who want this are likely very, very wrong and that this "feature" will turn out to be a footgun and a trap more often than a sniper rifle...

Why? It's just exposing the partitions underneath (which all horizontally scaled event/log systems use) and letting you order by the key within each partition.

This is exactly like Kafk, Pulsar, AWS Kinesis, Azure Eventhubs, and many others which are all being used by thousands of applications without issue.

Actually, it's not exactly exposing the partition underneath. The ordering key is the only unit a user has to deal with: think user ID or database row primary key. Throughput is limited to 1MB/s per ordering key, but the number of ordering keys is limited only by what can be represented in a 1KB string. One can have millions of keys if desired and they don't need to be known in advance.

The system scales automatically as needed, so there is no need to think in terms of partitions or worry about repartitioning when load increases. Order is preserved through this scaling without the user having to do anything.

Disclaimer: I work on Cloud Pub/Sub and am the primary engineer on the ordering feature.

It's hashing the ordering key to map to a partition right?

That's what I meant, access to ordering via partitions but without the full addressing of named partitions like Kafka.

There is a mapping from ordering key to internal storage structures, yes. The goal is that users are abstracted away from any notion of partition and the system handles scaling and ordering on its own, e.g., when there are hot shards.

Elaborate please, why is receiving messages in an ordered fashion a foot gun and a trap? Isn’t it totally acceptable to rely on this type of behavior if your use case demands it?

Because this will break in situations where you have multiple consumers or if there's any sort of redelivery/failure scenario.

Ordering only works if there's a single-threaded consumer somewhere that processes exactly one message at a time and blocks processing as soon as there is an error of any kind. The amount of scenarios that this works for is probably very small.

The footgun is that people don't realize the above constraints and assume that ordering will magically fix a lot of ordering issues (some more examples of this in the rest of this comments section).

One way it can be a foot gun is:

To ensure ordering, systems like these have to block consumption on any given partition until it gets an ACK. If publishers of the data don't understand that, they can publish with too coarse a partition key which greatly decreases the amount of concurrent consumption you can have.

Not knocking how these systems work, it's more that people generally don't "read the fine print", especially these days with cloud services.

Technically I agree, but I think in most cases, the alternative is the endpoints need to handle unordered messages correctly and that won't happen either.

So which is better: The queue does it, hiding subtle footguns because doing distributed systems reliably is hard; or the queue doesn't do it and the applications hanging off have failures more often (but not often enough for people to fix) because distributed systems reliability is still hard.

All of the uses of Kafka I have encountered in my professional career have been deployed simply to make something asynchronous, that is, kafka exists to be a low-impedance sink of things to do later in some other process. I've not run across one that actually needed those items to be processed in order. Therefore it is my impression that ordering can be a false requirement.

Agreed, also "processing in order" != "consuming from the sink in order". If everything upstream and downstream of the sink doesn't also happen in order then using an ordered sink is pointless.

Head-of-line blocking has two aspects: crashes and long processing times. If the first message for an ordered key can't be acked because it crashes subscribers, we can't make progress on the messages for that key. If the first message, likewise, takes too long, then it delays processing of subsequent messages for that key.

Even without ordering, we still have to deal with variations of these issues. A message may still crash subscribers, be redelivered, and crash yet more subscribers. A message may delay the processing of other messages on the subscription.

Here's one way to deal with crashing messages: keep a side-cache of processing attempts for every message ID and ack those that have been processed too many times before processing them. Cloud Pub/Sub has a dead-letter queues feature to do this automatically. (It is not yet implemented for subscriptions with ordering keys - one can't enable both features.) We can do something similar for long messages - acking the message on timeout or processing more messages in parallel.

So, with ordering keys, as it stands, you do have the extra problem of dealing with messages-of-death (crash-inducing messages) yourself, but the long-processing time issue still exists in a slightly different form. Hopefully, we can add DLQs for ordering keys if there's sufficient customer demand.

Disclaimer: I work on the Cloud Pub/Sub team, but this is my own explanation.

Reply on news.ycombinator.com