Hacker News Reader

What's wrong with Pachyderm? Sounds like you are having a lot of pain with them.

I am not affiliated at all with Pachyderm, I am just curious about this problem-space.

There is a good number of specific reasons, but it really comes down to this : they have too many features.

Is something like dolthub [0] what you have in mind?

No. They embraced the "Git for data" mindset, like Pachyderm and DVC. This design is restricted by git semantics when we need to go further.

Think more about Kafka's topic/message objects, with each message being an entire table, and their Stream API

Maybe a JCR implementation like Apache Jackrabbit Oak would be a good tool for this data. See here for sql semantics: https://jackrabbit.apache.org/oak/docs/query/grammar-sql2.ht...

Great comment! We're pretty much in agreement with all of this.

> can I query the same dataset at two different PIT using layered querying in SQL

Yes. We have a query that does this on our home page (it's the second in the carousel at the top):

    SELECT
        new.domain AS domain,
        new.id AS id,
        new.name AS name
    FROM
        "splitgraph/socrata:20200809".datasets old
    RIGHT OUTER JOIN
        "splitgraph/socrata:20200810".datasets new

This is querying two versions of the same image.

> are you storing the entire dataset as new or only the diff between versions

Basically we store the diffs between versions, yes. The simplified explanation is that data images are an initial snapshot + chain of diffs ("delta-compressed"). You can read more in the concepts section of the docs. [0]

> Git-like semantics (pull, push, checkout, commit) are poorly suited for versioned, immutable datasets

We pretty much agree with this. Some similar projects take the wrong approach and try to implement 100% of git commands "for data." But it's a square peg trying to fit into a round hole. The use cases are not the same. There are some similarities, but they're limited. Our goal was to step back and think from first principles: What do we like about our coding tools? How can we get the same benefits with data?

> Versions should be at least partially ordered and monotonically increasing. Hashes doesn't convey the information necessary to decide if dataset:de4d is an earlier version of dataset:123a, or not.

Sure. If you want monotonically increasing versions, you can do that by tagging an image with `sgr tag`. (FWIW, image hashes are currently random, but the hashes of objects that comprise them are not -- they're actually content addressable representations of the changeset from the root to that version using an algorithm called LTHash.)

> Tracing a derived dataset provenance will only work if you can assert that the "code" or transformations applied to the original dataset is deterministic (side-effect free).

Yes, definitely. Dataset provenance works best when you use Splitfiles [1] to construct images. It's still possible to run into problems with non-determinism, but for the OLAP use case, we think those situations should be rare.

[0] https://www.splitgraph.com/docs/concepts/images

[1] https://www.splitgraph.com/docs/concepts/splitfiles

Ah ! I initially missed the splitfiles page ! Declarative, cacheable... It's a great approach you have here. I think you could go even further and infer some SQL transformations from the schema definition.

Anyway, I'm curious to see where your project will end up. You put much more efforts in the design than most "for data" tools !

Cheers !