Can Kafka be thought of as a non-hosted version of Kinesis? I thought Kinesis would be a good solution to dump logfile data to for processing and ingestion. Could you explain some technical reasons to use Kafka vs. Kinesis? Thanks.
I believe Kafka can still lose some data if all the active machines fail. It's a deliberate design decision (it's the right thing to do if you want to remain available and can tolerate some data data loss). I believe the Kafka team are working on it, but it's non-trivial to fix.
If you process that much data, Kafka is one of the last things which you'll need to scale out.