KIP-1150: Diskless Kafka Topics

cwiki.apache.org

32 pointsenether3 comments

I presume this is part of the knock-on effects of Confluent (managed Kafka) buying WarpStream (Kafka emulated on S3 object storage).

Also a shout-out to Bento, the fork of Benthos after RedPanda acquired it.

Somewhat. The buying further validated the market, but at the time of purchase (Sep 2024) there already were 5 alternative proprietary solutions utilizing the same direct-to-S3 design shipped or announced (Confluent Freight, Bufstream, AutoMQ, Redpanda cloud topics, StreamNative Ursa). The cat was out of the bag that this is a great architecture

Full disaggregation of compute and storage is the right direction. Let storage handle replication, it's getting good, global, low latency, cheaper (like with S3 Express). Kafka becomes a smart data ingester and router: it moves bytes, enforces ordering, does minimal buffering. That's it. Do one thing well.

You get a system simpler to operate, to scale, and more flexible; data could be consumed outside of Kafka itself (in a batch way typically), without duplicating the data, that's a big win.

Reply on news.ycombinator.com