Comment by tiew9Vii | Hacker News Reader

Comment by tiew9Vii | Hacker News Reader

Parent Full thread

They offer commercial support services/contracts for Apache Kafka.

They sell proprietary Apache Kafka add-ons for enterprise such as authentication/authorisation RBAC. The open source Kafka release supports auth/authz but not consistently between the suite of tools. They also have a monitoring UI etc.

They release the commercial add-ons under the Confluent platform distribution.

They heavily push ksqldb which is a sql layer on top of Kafka that competes with https://lenses.io/. Confluent market it as a DB but that’s pushing the creative marketing.

They provide a hosted Kafka service.

The hosted service is likely more attractive to SMEs. It also has competition with the SME market with AWS providing managed Kafka. Confluents service does have extras on top of AWS like tiered storage behind a non open source license but nothing is stopping any one building there own tiered storage. I can see people using Confluent’s cloud but don’t see it being big.

Large corporations will (have) generally hosted/run their own Kafka.

I like Kafka but can’t see how Confluent can be profitable.

Cloudera is an example, they couldn’t make the service business work, I feel Confluent will have the same issues.

Elastic search have issues with AWS eating some of the market. Confluent have the exact same issues.

The Kafka is your nerve center narrative, that is great from Confluents perspective financially but it’s no different than the old enterprise service bus as the nerve center for everything. There’s a reason we don’t see that anymore.

AWS managed kafka is great but it's really just Kafka. Confluent offers stuff like topic monitoring, an http bridge, and other services that make kafka more practical to actually use (dare I say, missing features.)

My company uses Kafka extensively. We hosted it ourselves but after a few Kafka meltdowns we decided to go with a hosted solution. We compared AWS's offering to Confluent and at the time found that while Confluent had all of the nice tooling, we'd already built our own tooling to do the same thing, and rebuilding our entire infrastructure was a non-starter. So we went with AWS. It was a drop-in replacement for our own Kafka and it hasn't blown up on us since.

If we where starting fresh, Confluent would be more viable. I can't help but wish that the offerings that are part of Confluent where part of Kafka though. Kafka's lack of the tools that Confluent offers are imo what's holding it back from totally taking over the world.

We build a complete drop in replacement for Confluent's metrics/management tooling (https://kpow.io) that fills a very large gap in the engineering experience for Kafka. I say replacement but tbh we offer a lot more in terms of features other than stopping/starting clusters.

It feels obvious to me that AWS will roll-out S3 backed storage and managed Kafka Connect at some point, their recent IAM / ACL integration points to a pretty active MSK team.

I completely agree in the past there were some pretty big gaps - that's why we built kPow - my feeling is those gaps are narrowing pretty significantly.

Kafka S3 dumping would be amazing, could really clean up our tooling. I'll check out your product as well.

Honestly I can't wait for perpetual topics backed by S3 integrated with MSK. If the AWS team aren't working on it I will eat my own shoes.

Let me know how you go with kPow! If you want a guided tour just let me know. :)

ps. we're releasing a major bump to the UI later today..

> I like Kafka but can’t see how Confluent can be profitable.

I've seen what some corporate customers pay for an enterprise license. It's kind of insane.

SaaS S-1s often have a section about this. From Confluent's:

"As of March 31, 2021, we had 561 customers with $100,000 or greater in annual recurring revenue, or ARR, across a wide range of industries, compared to 374 such customers as of March 31, 2020, representing year-over-year growth of 50%. As of March 31, 2021, we had 60 customers with $1.0 million or greater in ARR, compared to 33 such customers as of March 31, 2020, representing year-over-year growth of 82%."

Yes it is yet they have massive losses. The money is also likely in the same ballpark of Cloudera / Horntonworks and other companies providing professional services for open source software who have all struggled to make the business model work.

We considered confluent at my last job, where our throughput was going to be miniscule to start but we required VPC peering...that bumped us into an Enterprise plan for something like 80k/year! Went with Aiven instead which was more like 20k/yr IIRC.

The trouble is if you need VPC peering then it requires a dedicated VPC, dedicated k8s, dedicated instances. Confluent currently only offers one instance size, so that dedicated offering can't be scaled down.

PrivateLink allows for shared VPC and hypothetically a shared k8s. So the cost structure can be better on the Confluent side. Of course scaling down instance types would make a huge difference, but I don't think that's supported currently. The cheaper option is multi-tenant Kafka clusters.

In general, I think VPC peering is a dead end for use cases where you are connecting two companies networks together. It's not great operationally and puts a lot of security / filtering burden on each party.

PrivateLink is a much better option.

I'm not too knowledgeable about all this, I'm sure you're right, and it was someone else's requirement that we have peering. Still, we'd grown accustomed to other vendors offering it by default (Timescale Cloud, Aiven) so it was a serious sticker shock and a big part of making us go elsewhere.

Their consulting when we asked was on the order of $6000 per consultant per day. And they were fully booked.

My beef with their tiered storage is that Confluent gets all the benefits while the customer gets almost none, at least with their hosted service: if you let them offload older messages to S3, they don't charge you any less for them! Can still get perf benefits, though.

> hey heavily push ksqldb which is a sql layer on top of Kafka

Can Kafka replace a database? We have an architect within our company that heavily pushes using Kafka for everything (as a database, a caching layer, message bus etc.,)

Kafka can replace a database in some cases, depending on your query patterns and ability to partition the data intelligently.

It excels at write-heavy workloads and, if you're already using Kafka for streaming or message passing then you can use it as a key-value store in order to avoid the extra operational burden of dealing with a distinct service for caching or storage.

But if you need a data warehouse or just a relational query engine for data at rest, or if your data volume is not too large and you're not already using kafka, you're probably better off with a regular DB.

Thank you.

Reply on news.ycombinator.com