Hacker News Reader

Comment by benjaminwootton | Hacker News Reader

I think there’s a huge unacknowledged gap in the database industry for good streaming products.

If we are building a report or dashboard that we pull up a few times a day then a pull based model where we query the database on page load is fine.

For almost anything else such as an app, a microservice, an alerting system, a web page, a dashboard, we want to be able to update it in near real time for the user experience. Receiving a stream of query results is by far the easiest way to do this.

Polling is obviously a poor interim solution.

I think streaming will be a huge story in data over the next decade. The products are coming through now which is a start.

View on HN

Materialize gives a vaguely PostgreSQLish user land on top of this idea. Obviously Postgres has LISTEN/NOTIFY which are fine for most places you need a queue and push updates but the key is incremental view updates so you get instant results on complex queries.

Unfortunately, Materialize has a pretty restrictive license.

https://github.com/MaterializeInc/materialize/blob/main/LICE...

I'm generally pretty happy to pay for open source software, but licensing like this is just too risky. I need to be able to experiment with something at scale, in production, before I start paying someone.

Free for 1 node, paid after. Code reverts to permissive license after 4 years. Think this is fair. you can bootstrap for free and only pay if it works out

It's more than fair, of course. It's just a barrier.

There's https://www.risingwave.dev which is similar to materialize.

I wouldn't quite say that this need is unacknowledged, most DBs have some kind of streaming interface for actual data contents (not queries, though), which for instance are leveraged by Debezium to expose change event streams in one uniform event format (disclaimer: I work on Debezium). Going foward, I expect that these capabilities will be substantially built out, also providing streaming query support (incrementally updated materialized views), not being locked in separate licensing layers or a more or less complex to use after-thought, but a primary part of database offerings, just as SELECT today.

Streaming has been the story for the past couple of decades already. But once you accept the value of an event-first architecture, the benefits of traditional RDBMSes become much weaker.

It’s been discussed for a long time, but the ecosystem is still spotty. Many databases have some kind of “live query” feature for instance but with limitations or not intended for production use.

A lot of work has taken place in the Kafka, Flink, DataFlow ecosystem but that still leaves a lot of work for the developer over a simple subscribe to query results.

I do think a lot of work has been done, but it all needs to move up a few levels of abstraction.

> A lot of work has taken place in the Kafka, Flink, DataFlow ecosystem but that still leaves a lot of work for the developer over a simple subscribe to query results.

Personally I find it much easier to write the code explicitly than try to understand what a query planner is doing, especially if performance is relevant. (That's not to say there's not plenty of room for improvement in the streaming world - but I'd rather have a helper library that I can use on top of the low-level API, than have to go through a parser and planner for every query even when I know exactly what I want to do) But I seem to be an outlier in this regard.

Yep. Exactly this.

Our focus (Hasura) is on the last mile so that innovations on the data side (eg: materialize, ksql, timescale continuous aggregates) are “just obvious” to start building applications and services against.

maybe if you're a really big company with petabytes of data but I wouldn't' say its worth the operational complexity for 90% of tech companies out there. Seriously, design your schema so you can have tables optimized for aggregate functions. you can use materialized views or have them populate via triggers. Either is still going to be an order of magnitude less work that dealing with all the edge cases that are ging to come up when you try to build a distributed data pipeline. Thats a death by thousand cuts for any small to medium startup that doesn't already have a team of engineers experienced in making such systems work.

Plus you can stream db changes from postgres to kafka for those edge cases where you really need it

TLDR: if you're in a startup and thinking of building a distributed system.. DONT. stick with a monolith and spin out services as NEEDED.

> maybe if you're a really big company with petabytes of data but I wouldn't' say its worth the operational complexity for 90% of tech companies out there. Seriously, design your schema so you can have tables optimized for aggregate functions. you can use materialized views or have them populate via triggers. Either is still going to be an order of magnitude less work that dealing with all the edge cases that are ging to come up when you try to build a distributed data pipeline. Thats a death by thousand cuts for any small to medium startup that doesn't already have a team of engineers experienced in making such systems work.

Funny, my experience is the exact opposite. Materialized views and triggers are death by a thousand cuts with all their edge cases. Whereas if you just use Kafka for everything from day 1 then everything works consistently and you have no problems.

So who's managing the kafka cluster? are you running scheduled backups of it? what happens if the system is down while ingesting data? I can only imagine you have someone thats already veery experienced in kafka if you find that easier to manage than an rdbms which already has established hosted services available.

Running Kafka is fiddly but a lot easier than running a true master-master RDBMS setup (which none of my employers have ever managed; at best some have had a read replica and an untested shell script from five years ago that supposedly promotes that read replica to master if the master goes down). Backups are the same problem either way (and, while it's not a full replacement for actual backups, the fact that Kafka is clustered means you're protected from some failure scenarios out of the box in a way that you aren't with something like Postgres). And there are plenty of established hosted kafka services if that's your preferred approach.

Would you not be able to solve this by having a messaging queue and getting the database to publish it’s updates?

Curious what products you’ve seen as well.

A message queue is a database too.

So the question is: Why have two copies of your data, two products to learn and monitor and operate, write boilerplate to move data between the DBs, etc.?

A "message queue" comes down to being another index on your table/set of tables ordered by a post-commit sequence number. These are things all SQL DBs have already, it just lacks a bit of exposing/packaging to be as convenient to use as a messaging queue.

That's why I was watching rethinkDB but that didn't pan out.

Reply on news.ycombinator.com