Given this, why do people deploy it to AWS? It seems like an invitation to disaster.
Given this, why do people deploy it to AWS? It seems like an invitation to disaster.
It's a pity, because it would be great to have an open source component for the Kafka architecture, but where data-loss is unacceptable. Hopefully this will be fixed in Kafka itself; I've also been working on something that is based around Raft.
http://docs.aws.amazon.com/kinesis/latest/dev/service-sizes-...
As far as the API itself though, as is usually the case you might need to get a bit creative to design around AWS imposed constraints due to their design decisions. It might end up not feasible, or not worth it, for your particular use case. But I believe it's at least worth investigating/discussing.