Fast, Reliable, Stream Processing - Flink
Data Science on Big Data - Spark
Reliable Distributed Log Aggregation - Flume
Low Code Data Flows w/ GUI - NiFi
Zapier for Enterprise Software - Camel
A Single API for Batch and Stream Jobs. Execute on Spark, Flink, managed services etc. - Beam
Never heard of anyone use Apex over Flink. Storm community has branched off into Flink, Spark Streaming, Heron and Cloud Dataflow.
Explaining Spark vs Flink is quite hard tho.
Spark was built as a Batch job solution. Flink was built as a Streaming solution. Since Spark needed to adapt, they leveraged micro-batching to operate as a streaming solution.
They are very similar today. Maybe some remnants of the original design make Spark "slightly less appropriate" for pure streaming usecases, or possibly less able to iterate on future features/optimizations, but so-far that line of reasoning has been only speculation.
- Flink guarantees exactly once stream processing through a barrier checkpointing system.
- Flink has very detailed APIs for handling state and doing stateful stream processing.
Additionally, Flink has one of the most active communities in the Apache Software Foundation: https://blogs.apache.org/foundation/entry/the-apache-softwar...
In situations where you want to create a streaming application with persistent state you might choose Flink over spark streaming.
We use Camel, Spark and Beam to do much the same thing at my work, so despite using them all semi regularly, I've never known why you'd pick one over the other.
[1] https://github.com/gunnarmorling/awesome-opensource-data-eng...