rmb938

That's not really a M.2 nvme to sata converter though. That's just something that can take pci-e lanes and either give them to nvme drives through a pci-e multiplexer or convert the pci-e lanes to saas or sata. It also isn't passive, there's lots of processors to make it happen.

Maybe it's worth trying out GCP for a POC cluster? Downside is they don't have any ARM instances but some back of the napkin math does show that an equivalent setup of using 6x im4gn.2xlarge in AWS to 6xn2-standard-8 and 6x3750gb pd-balanced ssds is roughly around the same cost and disk perf, could be a bit cheaper with their AMD instances instead of Intel. If you compare it to a gp2 disk it's roughly 4x faster, but the same per for local disk on im4gn.2xlarge.

I also have had decent success with getting committed use and reservations on GCP ssd compared to other cloud providers.

Maybe I missed it but are you able to talk about how many messages a second, partition count and average message size?

I run a few hundred Kafka clusters with message counts per second in the tens of millions for some clusters, a few thousand partitions, message sizes around 7kb with gzip compression, and have never needed the amount of CPU and network/disk throughput mentioned. With node counts range between ~10-25. Most of my clusters reaching those speeds at most average around 7Gbps of disk throughput per broker.

I have recently started running Kafka in GCP with their balanced ssd disks capping out at 1.2Gbps I'm not seeing much of a performance impact. It requires a few more brokers to reach the same throughput but not having any of the performance and scaling issues mentioned in this post.

My brokers are sized a bit differently than mentioned in the post as well, low amount of CPU (maximum 20ish cores) but much more memory around 248GB for my larger clusters. So maybe that has to do with it? Maybe the broker sizes that were chosen are not ideal for the workload?

Maybe I've been lucky in my setups but I would like to know a bit more. Having been running Kafka since the 0.10 days and now on 2.6 for all my clusters this type of performance problem seems a bit puzzling.

For those that are not too versed in distributed systems, kubernetes or docker I highly suggest you read the comments on the post. I am going to promot myself a bit here and link directly to my comment https://medium.com/@rmb938/thanks-dan-for-the-great-post-1ef...

Sadly a large environment was brought down due to not following best practices. This event probably could have been prevent or at least have very little impact if best practices were followed.