Hacker News Reader

Comment by ramraj07 | Hacker News Reader

I am squarely in the "people interested in building an event based architecture". Not a CS background, but know tech decently. I typically know the names of more of these apache projects than most people I've talked to IRL (though ostensibly I'm not part of the tech elite). Yet pulsar I only came across on HN a week back. And nothing changes due to the knowledge. It's yet another apache product which has great design but needs either a genius or an army of devops to deploy. Like even if I figure out what the hell this thing is, I'm then tasked with figuring out what the hell zookeeper is (like for real, I'll buy a beer for someone who can successfully ELI5 wth zookeeper is. And also pig. Or impala).

View on HN

If you can't understand what Zookeeper is, I'd recommend reading Martin Kleppmann's book Designing Data-Intensive Applications (https://dataintensive.net/).

You don't need a CS degree to work in this field (I don't have one either!) but there are fundamental concepts you need to understand in order to make informed decisions when designing distributed systems.

You don't need a CS degree or Martin Kleppmann's book to work out it's a GPITA.

I am also not a fan of Zookeeper but I have come to respect it. In defense of Zookeeper, distributed systems are a GPITA. If you need to select some components why not go for ones that are solid [1].

[1] https://aphyr.com/posts/291-jepsen-zookeeper

I'm no expert, but my understanding is that pig is a combination of

- a language for specifying data transformations, and

- an engine to compile programs written in that language into mapreduce jobs to execute on a hadoop cluster

it was designed to easily map some common functional and SQL idioms (e.g. filter, group by w/ aggregation functions) to parallel execution for processing huge amounts of data.

Impala is another big data project that is an engine for planning and executing SQL on data stored in a hadoop cluster.

Zookeeper is... black magic??

> Yet pulsar I only came across on HN a week back.

Same - was it the comment saying nobody chooses Kafka anymore? I was surprised by it.

If you search you will find that there are always the same 4-5 HN users who comment on pulsar/kafka topics arguing that Kafka is not a viable solution anymore. Even though almost everyone is still using Kafka

Meanwhile the Confluent Kafka conference seems to double in size year after year. And AWS chooses to offer Kafka as a service, rather than Pulsar. These "<Solution> is dead / not viable anymore!" comments are pretty rampant in the Big Data space. Some are true though but it still makes me roll my eyes.

These people seem to be invested (quite literally) in Pulsar disrupting Kafka's position in the pub/sub & event streaming space. This isn't to say Pulsar can't someday be superior, though. Time will tell. But Kafka has such a massive lead and it fits the needs for most people who need to do data integration that I don't see the need for everyone to uproot their solutions in favor of Pulsar.

Almost everyone still uses Microsoft Windows.

But almost everyone has heard of GNU/Linux or Apple OSX.

pulsar was seeing quite a bit of hype back in ApacheCon 2019 already

Apache Pulsar should not be confused with another quite popular Pulsar:

Event driven concurrent framework for Python https://github.com/quantmind/pulsar

Reply on news.ycombinator.com