Hacker News Reader

Comment by sbr464 | Hacker News Reader

Curious if a potential solution would be having open, read only databases, that you could query directly, vs everyone copying the same data over and over. Kind of how you don’t download Wikipedia but access what you need. I realize there are a lot of things to consider. But not even a rest api/etc, an actual database.

Realize it wouldn’t scale, would cost money etc, but could be interesting

View on HN

I've been thinking about the same for quite a while now. In fact, look at the overlap between GraphQL and SQL conceptually. I absolutely think there is something to this.

In the past, I have used certain wide open read only genomics databases (not going to name it so it doesn't get hammered by HN).

Other posters are right about services such as BigQuery but I think there's a place for an open source project here that interfaces SQL to databases through a layer that adds caching, throttling and more services on top of that. That's how you make it scale.

The Dremio project (open source by the backers of Apache Arrow) has a SQL REST API that converts a standard SQL dialect/datatypes to the underlying systems. I think that's a good start and Dremio has a ton of other awesome functionality like Apache Arrow caching.

Simple model is expose an expression language (even could be not SQL, like jsoniq, or other expression languages), mapper from that to SQL, web service API on top with a pluggable connector model.

I say that I'm going to start an open source project around this all the time but haven't gotten the inertia to do it. Argh!

Google BigQuery does this though. They host huge public data files and then only charge for the queries.

Your right, I've tried it before and forgot about it.

Also 'requester pays' on Amazon S3 buckets.

This is the kind of application that dat, swarm, ipfs/filecoin are aiming to support.

More specifically orbitdb [1] which is built on top of ipfs is a potential starting point. It implements a decentralized and distributed append only log data structure as well as object store and key/value lookup db. Not quite a SQL database though.

[1] https://github.com/orbitdb/orbit-db/blob/master/README.md

How about using a database engine with built in horizontal sharding to distribute a big database to multiple users in smaller pieces, just like bittorrent but with query-capabilities.

Thanks to the recent big-data nosql craze this should be easy to find off the shelf. As example i know MongoDB has all this built in, not sure it is designed with untrusted nodes in mind but otherwise fits all requirements.

I think cockroachDB has the automatic node registration/balancing? I guess you’d have to deal with the shard or data you would choose to host

There’s one feature of Faunadb that stood out, the ability to create a seemingly unlimited number of nested databases, where the db assigned to you appeared to be the root/full database, with full query/write privileges.

Imagine a public dataset that you could query but also update with your own findings and metadata, and choose to have it shared upstream or keep private. The engine could diff your work to avoid duplicating a copy of the data. You could sync it locally for speed if needed

Thanks for the resource though! Looks interesting

That’s what the current academic publication infrastructure was built to be. Then capitalism happened, and we have abusive monopolies instead.

Historically, as empires collapse, they burn their libraries, but the usually don’t get to the copies that are kept on the outskirts.

For sure, I was thinking more simply how most of us developers etc, immediately create an abstraction (rest/graphql/sdk) vs allowing direct queries to a db. Would be interesting if libraries/knowledge/datasets could be a query away, actual free info, not a search engine/promoted/abstraction in any way.

Reply on news.ycombinator.com