Hacker News Reader

Comment by virgilp | Hacker News Reader

> Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing.

Well, no, you went too far.

Kafka is primarily used for communication & data transport, by most people (can be used in other ways, and it has the KafkaStreams library that enables you do to some computation on said data - but it is, primarily, a transport & communication mechanism; also maybe storage, if you squint right)

Spark and Flink might be similar on first sight, but if you look a bit closer you realize Spark is primarily geared towards batch workloads, and Flink towards realtime. Sure, you can do micro-batch in Spark and pretend that's realtime stream processing, but the focus of it is fairly clear - as is the focus of Flink. So both have legitimate rights to exist.

I'm not sure about the others, haven't used them. There may be indeed consistent overlap - but I'm sure they are different approaches. What's wrong with that?

View on HN

On the contrary, I didn't go far enough. I didn't talk about Apache Gearpump, NiFi, Beam, Ignite, or Trident.

I know there are subtle differences in each specific technology, and that's probably the same justifications used to support the thesis when building yet another very similar framework.

> What's wrong with that?

I believe it drastically reduces adoption of these tools because many of us avoid what appears to be bandwagon technologies as we don't want to consciously add layers of future technical debt when a majority of these projects will be abandoned.

I said "you went to far" to claim Kafka, Spark and Flink do basically the same thing. It's reasonable to use all 3 of them in the same team - so they clearly don't do the same thing.

> I believe it drastically reduces adoption of these tools

As does any competition. Tons of smartphone makers = less adoption for any one of them - and many will close down. Still not a bad thing.

> As does any competition.

Those are open-source frameworks, which is antithetical to competition. The whole point of open-source is that the contributions of one entity can benefit everyone, while in this situation, having so many frameworks "dilutes" the efforts because the same problems need to be solved in each framework. Smartphone makers aren't here to share their technical advances with everyone, they're here to make money.

I'm not saying we should purposefully agree on killing all but one framework, but the ideal situation is if one or two come out at the top as being just the best in their category so that everyone just use them, and all efforts converge towards those.

I don't think that's how it works; "best" is sooo hard to define. Humans are very diverse, and have different needs/preferences. Also what is best today may not be best tomorrow (not because of the system itself, but because of changes in the environment), so it pays off to have "sub-par" systems evolve in parallel with different slightly different goals or implementations.

I don't see "open-source" as being antithetical to competition, at all. In fact, e.g. Databricks only exists because of Spark. There is competition in the open-source world too, and IMO that's a good thing.

Since when is there no competition and diversity in open source? Isn't too much choice usually a major complain with linux, programming languages, frameworks, etc?

Nobody said that. But having too many competing overlapping products does mean more fractured effort spread amongst all of them.

>I'm not saying we should purposefully agree on killing all but one framework, but the ideal situation is if one or two come out at the top as being just the best in their category so that everyone just use them, and all efforts converge towards those.

That assumes you can have a single framework with all the features people want without causing issues (conflicting basic designs, high cost to maintain, deployment costs, configuration costs, etc.).

To be a bit more future-proof you should give Apache Beam a try. The same code should (theoretically) work with any of the supported runners[0] and so you could deploy it on top of the most suitable framework/technology for your specific workload. Moreover, at this point, the community has several examples of how to add an additional runner.

Edit: I didn't see you mentioned Beam in your second salvo :)

[0]: https://beam.apache.org/documentation/runners/capability-mat...

In the case of Spark and Flink, I wouldn't say that batch processing versus realtime stream processing are "subtle differences". That's akin to arguing that relational databases vs. document stores vs. timeseries databases just "muddy the waters".

Hacker News and Reddit have a lot of interesting discussion. But the audience skews toward client-side webdev, and students or younger developers. An audience accustomed to libraries and frameworks that you can reason about with fairly low learning curve, and spin up in a Codepen to see visually right away.

Heavy-lifting server side tools, especially those who only earn their keep at scale, are a different beast. And that's OKAY. Quite frankly, if you're "not sure" whether you need a stream processing platform in your architecture, then YOU DON'T. Aside from some consultants and salespeople, no one's really going to push you toward adoption of this stuff.

In the overwhelming majority of use cases, what you need is a tiny microservice (in your language of choice). Which reads from a Kafka or Rabbit topic, and stores state in your cache system of choice. By the time you reach the scale where that's not suitable, your organization probably won't need a web forum thread to educate you on what the vendor landscape looks like.

>I believe it drastically reduces adoption of these tools because many of us avoid what appears to be bandwagon technologies as we don't want to consciously add layers of future technical debt when a majority of these projects will be abandoned.

Doesn't seem to be impacting adoption of the major ones from what I can see (Spark, Kafka, Flink). Assuming you're at a scale where you need these technologies. Which most companies are not so it's a good thing they don't implement them just for kicks or as "future proofing." If you have a business problem that the main frameworks can't solve then trying out some of the other ones may be worth the cost. There's also ongoing support for older frameworks (Storm) so it's not like your code becomes useless.

The approach open source is taking to resolve this issue is to embrace the diversity but create unified APIs on top of it. Apache Beam and Arrow for defining data workflows and data exchange formats respectively. There's also always SQL which works with a lot of the more data warehouse solutions out there (with some tweaks per solution unfortunately).

> I believe it drastically reduces adoption of these tools

I really don't see why. From afar they might appear to perform the same task but once you take a closer look you quickly realize they follow fundamentally different architectures and have significantly deployment and performance characteristics.

Just because there are all kinds of hammers, but there are also plenty of different uses, and even in the subset involving driving nails there are significant requirements.

Reply on news.ycombinator.com