Hacker News Reader

Comment by wenc | Hacker News Reader

A quibble I have with the term "stream-table duality" is that it's not true duality.

You can construct a state (table) from a stream, but you cannot do the reverse. You cannot deconstruct a table into its original stream because the WAL / CDC information is lost -- you can only create a new stream from the table. This means you lose all ability to retrieve previous states by replaying the stream. Information is lost.

Duality in math is an overloaded term but it generally means you can go in either direction. This is not true here.

View on HN

It depends on your data schema. If you store the data in a temporal form (every row has a transaction period), you can reconstruct a stream with ORDER BY LOWER(transaction history).

One benefit of using the table as a source of truth this way is that you can run migrations on the entire “stream” if you will, and republish the events in order, in the new schema definition.

Another benefit is that you can run SQL queries against your entire transaction history. Some (imo sketchy) tools exist that try to give you a SQL interface to Kafka, but that’s just bad. It’s much easier to have your cake and eat it in postgres than it is in Kafka, imo. Unless your data volume is absurd, in which case Kafka is your only realistic choice.

Of course, in a “literal” sense you’re correct, it’s not possible. But from a practical point of view, if your bit empor all data implementation is stable, you can totally depend on your ability to produce streams from a database. We do this at my current place of work all the time.

Just came here to post this comment. I feel like they must've known this when creating the terminology, so I can only imagine they called it "duality" because they wanted it to sound cool. But it's most definitely not duality and I would wager calling it that is actively harmful. Last thing you want is someone implementing a table under some misled impression that it can do everything a stream can!

This Confluent blog is a rehash of earlier conceptual position statement of Jay Kreps, who is now the boss of the Confluent blogger.

https://engineering.linkedin.com/distributed-systems/log-wha...

Pointing out (GP and yourself) that this is an 'asymmetric duality' is informative and valid.

Thank you -- it's food for thought.

A symmetric dual is one where there is equivalence: the dual of the dual is the primal.

But it is possible to have a "weak" duality that is assymmetric, where the dual encompasses the primal -- where there are regions of equivalence instead of a 1-to-1 equivalence.

I'm not sure if stream-duality fits this conceptual mold exactly but thank you for providing a counterpoint to my assertion. I'm going to think on it.

If one computes in a bounded time domain, then it is symmetric: a table is a ‘time slice’ of a stream.

so I can only imagine they called it "duality" because they wanted it to sound cool

Also, that you believing it is the most profitable mindset - for them. Company blogs are marketing, first and foremost.

I read it as “there exists a event schema and some database table for which they can be co-constructed”, not “every database table can be turned into a stream of events.”

I don’t have a proof, but For things like state transitions, I’ve often solved this with a <new_state>_at column. Seeing that field populated would indicate a second event need be emitted. (Assumes no cycles in the state machine.)

Often that table is not the one you have, even if (perhaps especially if) you emitted events for every update — seems like some entropy-like force pushes the universe toward emitting details in events that aren’t reflected in the table records, over a long enough timeline.

Couldn’t they just make the stored data bitemporal?

SQL temporal tables?

Difference algebra over the deltas could find some interesting relevance.

I've heard Haskell programmers talk about Data and Codata, and came away with the loose sense that Data might be like state and Codata might be like a stream.

That one would so confidently assert that "Codata is the categorical dual of Data" has me thinking there might be something here?

Anyone want to tell me how far off I am?

Data and codata are dual, but that doesn't mean they are the inverse of each other. It means we can formalize them both in category theory and they have the same formalization with reversed arrows.

More usefully, data is a finite data type and codata is an infinite data type.

Some example:

    sum = foldr (+) 0
    sum [0,1,2,3]
    > 6
    sum [0..]
    ...loops forever...

    sum' = scanl (+) 0
    sum' [0,1,2,3]
    > [0, 1, 3, 6]
    sum' [0..]
    > [0, 1, 3, 6, 10...

Haskell doesn't actually distinguish between finite and infinite data. Some functions work fine on infinite data, some functions loop forever.

Induction describes how to process data without diverging - process each input element in a finite time.

Co-induction describes how to process codata without diverging - produce each output element in a finite time.

This is useful if you want to prove things about infinite processes like operating systems.

Reply on news.ycombinator.com