HNHacker News
TopNewBestAskShowJobs

timsehn

350 karma · joined February 15, 2019

CEO of DoltHub
submissionscomments
timsehn··on US Presidential Election $25k Database Bounty Review
CEO of DoltHub here.

The data created by bounties is free and open.

Our business model is to sell database licenses. We're a database start up. Bounties are the thing that shows off our capabilities the best, so we consider it marketing. It could be more in the future, ie. a two-sided marketplace, but right now, we're just getting started.

timsehn··on Launch HN: Replicate (YC W20) – Version control for machine learning
Thanks for the shout out here. CEO of DoltHub speaking :-)
timsehn··on Don't Panic
CEO of DoltHub here, just want to go on the record and say discovering Noms made us realize Git semantics on top of a SQL database were closer at hand than we presumed. It changed the direction we were heading and we like the place we ended up.

Sure, the panics sucked but it's over. Happy to figure out a way to get those changes back into Noms eventually. When we changed it, it was unclear anyone would ever work on Noms formally again.

timsehn··on Gretel is building the GitHub for Data
CEO of DoltHub here. Thanks for the shout out. The key here is that we also built an open source SQL database you can branch and merge. It's the Git in the GitHub.
timsehn··on Show HN: Splitgraph - Build and share data with Postgres, inspired by Docker/Git
Also a survey of other "Git for data" options from a few months ago: https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-...
timsehn··on Show HN: Splitgraph - Build and share data with Postgres, inspired by Docker/Git
CEO of the company that built Dolt here. I just did some preliminary research and it seems that we have a similar mission. We both want to version control data.

The thing Dolt does that splitgraph does not do is support branches, diffs, merges and conflicts on data and schema. Dolt has its own storage engine to do this efficiently whereas Splitgraph relies on Postgres.

Having native Postgres with a versioning layer on top has other advantages so we're excited to see how Splitgraph's approach works in practice. Excited to play with it. We love to see more tools in this underserved space.

timsehn··on Recently minted database technologies that I find intriguing
CEO of Dolt here.

You are indeed correct, we are quite young. We are also focused at this stage on the data distribution use case (ie. use Dolt instead of putting a CSV or JSON file on GitHub). So, we haven't spent much time on query performance or scale. The current model is that the database and all of its history have to fit on the user's hard drive. The format is easily distributed (ie. put content addressed chunks on different servers, optimize for fewer network hops) but that's not how it works today.

That being said, we think architecturally we can eventually get pretty close to parity with other RDBMS on the read path. We will be slower on the write path given the need to build the Merkle DAG on writes.

Long ways to go though, we only launched open source last August. On a suite of MySQL ~6M correctness benchmarks, we currently execute 3-4X more slowly. These aren't large sets either so we suspect we'll run into some non-linearities in performance. This is just normal SQL. We haven't really tested the limits of how many branches we can handle or how long branch creation or merge takes at scale. Not because we don't want to but because it's not the use case we're focused on.

If you are willing to take the journey with us, we'll work with you to get Dolt where it needs to be (if it can, reads should be fine, writes will be more expensive). But we're just at the beginning so there will be bumps along the road. If that's not for you, we completely understand.

timsehn··on Cell History Inspection on DoltHub
That is not true. Dolt will throw a merge conflict if two cells are modified on two different branches. You still have to pick which one but at least you know.
timsehn··on We Need DevOps for ML Data
You have to move your data to Dolt. Dolt is a database. It's got its own storage layer, query engine, and query parser. Diff queries are fast because of the way the storage layer works.

Right now, Dolt can't be distributed (ie. data must fit on one hard drive) easily so it's not meant for big data, more data that humans interact with, like mapping tables or daily summary tables. But, long term if we can get some traction, we plan on building "big dolt" which would be a distributed version that can scale to as big as you want.

timsehn··on We Need DevOps for ML Data
Here's a list of companies/tools in the Git for Data space:

https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-...

timsehn··on We Need DevOps for ML Data
Tim , CEO of Liquidata, the company that built Dolt and DoltHub here. This is how we store the version controlled rows so that we get structural sharing across versions (ie. 50M + one row chgange becomes 50M+1 entries in the database not 100M with no need to replay logs):

https://www.dolthub.com/blog/2020-04-01-how-dolt-stores-tabl...

timsehn··on Using “Git for data” to find and analyze test regressions
CEO of company that built this here. You need a versioned SQL database in your toolbox.
timsehn··on How Dolt stores table data
Shout out to @aboodman who also just published his version of Prolly Tree documentation on Noms GitHub: https://github.com/attic-labs/noms/blob/master/doc/intro.md#...
timsehn··on Dolt is Git for data
We compared DVC and a bunch of other Git for data" tools here:https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-...
timsehn··on Dolt is Git for data
We're working on building performance benchmarks right now. We started with correctness. You can read about our correctness journey here: https://www.dolthub.com/blog/2019-12-17-one-nine-of-sql-corr...

We think over time (like years) we can achieve read performance parity with MySQL or PosgreSQL. Architecturally, we will always be slower on write than other SQL databases, given the versioned storage engine.

Right now, Dolt is built to be used offline for data sharing. And in that use case, the data and all of its history needs to fit on a single logical storage system. The biggest Dolt repository we have right now is 300Gb. It tickles some performance bottlenecks.

In the long run, if we get traction we imagine building "big dolt" which is a distributed version of Dolt, where the network cuts happen at logical points in the Merkle DAG. Thus, you could run an arbitrarily large storage and compute cluster to power it.

timsehn··on Dolt is Git for data
Very cool. Have not heard of this one. Will check it out.
timsehn··on Dolt is Git for data
Like "as of" queries or history queries? We have both of those.

AS OF: https://www.dolthub.com/blog/2020-03-20-querying-historical-...

HISTORY SYSTEM TABLE: https://www.dolthub.com/blog/2020-01-23-access-to-everything...

You can run `dolt q -r csv -q <query>` to output whatever you want to a CSV. We would need to do work to output a hierarchical format.

I'm sure it's possible to build whatever time travel operation you want. We can produce an audit log of every cell in the database pretty quickly.

timsehn··on Dolt is Git for data
You can make a solid case and I tried: https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-...

But there are some limitations for large tables. Diffs only work if you order the file in the same order. Plus, you have to import the data into some other tool to do anything with it.

timsehn··on Dolt is Git for data
Dolt is and always will be open source. Feel free to fork a copy and make it your own.
timsehn··on Dolt is Git for data
See the comment down below from a cofounder of Pachyderm :-)
timsehn··on Dolt is Git for data
We wrote a couple blog post on data licensing. We're not lawyers but we did a bit of research:

https://www.dolthub.com/blog/2020-02-24-data-licenses/ https://www.dolthub.com/blog/2020-02-26-copyrightable-materi...

timsehn··on Dolt is Git for data
No data diffs is the data is binary. But diffs are cell-wise like so: https://www.dolthub.com/repositories/Liquidata/corona-virus/...

git-lfs lets you put store large files on GitHub. With Dolt, we offer a similar a similar utility called git-dolt. Both these allow you to store large things on GitHub as a reference to another storage system, not the object itself.

timsehn··on Dolt is Git for data
I'll check it out. We think the world is a little more ready for this now, given how widely Git is adopted and the advances in other data tooling (like ML). But, as we're all aware, starting a business is hard :-)
timsehn··on Dolt is Git for data
We use AWS but will switch over to Google or multi-cloud when we exhaust our credits.

The system is still pretty simple. The main cost is the storage for the blobs in the Dolt repos pushed to DoltHub. We use S3 for that. There is an API that receives pushes and writes any other metadata (user, permissions, etc) into an RDS instance that stores metadata for DoltHub. That instance is also used to cache some critical things. Then it's just a set of web servers and a GraphQL sitting on top serving our React app.

timsehn··on Dolt is Git for data
The storage system we use only stores the rows that change. We have a blog post we're publishing on Wednesday explaining how.
timsehn··on Dolt is Git for data
You can export to whatever you like. In our imagination, data versioning would sit upstream of production just like source code versioning. You take the data out and do what you need with it in a "compile" step.
timsehn··on Dolt is Git for data
This blog compares us to Quilt and others: https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-...
timsehn··on Dolt is Git for data
You can spin up a MySQL compatible server using `dolt sql-server`. It does not implement everything right now but we'll get there.
timsehn··on Dolt is Git for data
We think this is the assumption (that data does not change) but we don't think it holds and data diffs help show you when something you did not expect did change.
timsehn··on Dolt is Git for data
You are correct. It is an homage to Git. We needed a word that meant "idiot" or similar, that started with D for data that was short enough to type on the command line.
← PreviousPage 2 of 3Next →