Hacker News Reader

Comment by jsmthrowaway | Hacker News Reader

Well, the big one is that Heron is built to support real-time streaming while Spark Streaming is not, given its choice to use micro-batching. If latency matters, that matters to you.

I'd appreciate if it were called Spark Microbatching, but we can't have everything.

View on HN

The amount of FUD thrown around about micro batching is really kind of silly.

How many streaming analytics use cases are ok with the JVM, ok with 10s of millis of latency, but not ok with 100s of millis of latency?

Remove "analytics" from your statement and you'll arrive at the answer, because while analytics is driving the streaming space as we speak it is literally one of the most boring applications of streaming imaginable. I positively cannot get excited about engagement numbers from an event pipeline. Intrusion detection, fraud detection, about half of infrastructure monitoring tasks, event sourcing... You're also glossing over the impact of microbatching which looks more like seconds than milliseconds, as well as what microbatching does to your windowing abilities (such as having to double-process to work around the microbatch interval as applied to your desired windowing semantics).

Microbatch latency immediately rules out several useful applications of streaming. I also didn't say anything about the JVM nor it being okay for my purposes. You did.

I can back up my statements on streaming from Flink to Samza to Dataflow/Beam to Storm to MillWheel to Spark "Streaming" and back again because it has been my primary focus (literally thinking of nothing else) for a couple years. Please accuse someone else of FUD because that's a relatively veiled way to say "you don't know what you're talking about," and I assure you that you're (condescendingly) wrong. I think you're also coming at that angle from interpreting me as negative on Spark Streaming. Read carefully.

> You're also glossing over the impact of microbatching which looks more like seconds than milliseconds

I've run spark streaming jobs at 250ms batch times. This comment you made, right here, is why I used the word FUD.

Okay, the better question then would be, how does this compare with writing a processor on top of Akka streams?

Reply on news.ycombinator.com