Just curious how does Kafka handle data rentention though? Can it be easily configured? Or you need to build something from scratch?
Kafka supports replication and fault-tolerance, runs on cheap, commodity hardware, and is glad to store many TBs of data per machine. So, retaining large amounts of data is a perfectly natural and economical thing to do and won’t hurt performance. LinkedIn keeps more than a petabyte of Kafka storage online, and a number of applications make good use of this long retention pattern for exactly this purpose.
From http://radar.oreilly.com/2014/07/questioning-the-lambda-arch...