> when consumers are lagging behind, producer throughput falls off a cliff because lagging consumers introduce random reads
I am confused by this. The format of Kafka's log files is designed to allow reading and sending to clients directly using sendfile, in sequential reads of batches of messages. http://kafka.apache.org/documentation/#maximizingefficiency