I don't have a good idea of the requirements at Spotify, but looks to me that using a streaming system like Storm or Spark Streaming would solve the 30 min event delivery delay they are experiencing w/ unstructured text -> Avro conversion. The latency for delivery would go down to sub-second levels.
Once persisted the consumers can just read kafka data and send them to hadoop, with less latency. Or you can plug storm o spark in as you said and do the analisys there real time. Or both.
I'm just intrigued why.
> When the system was built, one of the missing features from Kafka 0.7 was the ability of the Kafka Broker cluster to behave as a reliable persistent storage. This influenced a major design decision to not keep persistent state between the producer of data, Kafka Syslog Producer, and Hadoop. An event is considered reliably persisted only when it gets written to a file on HDFS.