My company offers a data feed over WebSocket that could reach 70MB/s if we allowed you to subscribe to the entire feed at once (we don’t). No one ever planned for it to handle that much data, but here we are. Because WebSocket servers do not scale horizontally we had to consume the entire feed on one machine. (Edit: ofc you can scale the # of clients handled, but not the data throughput!)

Initially our Go app could only manage reading 10MB/s from Kafka, which is pretty sad. After switching from Shopify’s “Sarama” Go library to librdkafka (a Kafka client written in C), as well as some other arcane optimizations, we were able to 20x our throughput and handle up to 200MB/s, despite the overhead of interop between C and Go. On top of this, we used a library that wrapped librdkafka in Go bindings (confluent-kafka-go) and it turned out to have a much simpler interface than Sarama, in addition to it outperforming Sarama 20x.

While we didn’t end up rewriting the thing in Rust, we did end up leveraging a library written in C for major performance wins, in a case where scaling out was not possible. And let us also remember that even when scaling horizontally is possible, it is not necessarily even desirable in all circumstances: distributed systems, aside from introducing some amount of overhead, are basically Pandora’s box wrt. failure modes.