HNHacker News
TopNewBestAskShowJobs

zh217

506 karma · joined September 21, 2016

submissionscomments
zh217··on Asami: A flexible graph store in Clojure
More precisely, Cozo currently has a RocksDB backend. The development version also has SQLite backend (for easier compilation on mobile platforms), in-memory backend, etc. It now can run as a web assembly module as well (see https://cozodb.github.io/wasm-demo/)

With respect to the SQLite backend, Cozo uses SQLite purely as a KV store with range scan capabilities, with both keys and values stored as bytes. This actually results in faster queries than offloading some stuff to the SQLite engine for the (very few) cases we tested.

Disclaimer: I'm the author of Cozo.

zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
Around 10 GBs of data, on the standalone version. We will have systematic benchmarks when the API, syntax, etc. settle down a little bit.
zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
Yes. For example, in Postgres you can sort tables arbitrarily large, not constrained by main memory. Postgres uses external merge sort when the tables are really large. There are other situations where the working data are disk-based when they are too large in Postgres. We will eventually be able to do that in Cozo as well, but no timetable is available yet.

For your second question, say you have a relation with lots of fields, one of them particularly large. As long as you don't use that field in your query, it will not impact memory usage. The query may be slower though since the RocksDB storage engine needs to read more pages from disk, but the fields that are loaded by RocksDB but not needed will be promptly evicted from memory.

zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
Technically, Cozo is using something called "Datalog with stratified negation, stratified aggregation, and function symbols", allowing aggregations to be auto-recursive when they form a semi-lattice, together with built-in algorithms which are black boxes taking in relations and giving you back another relation. Your example is taken care of by the "stratified negation" part.

I believe "Datalog with function symbols" is already Turing complete, but you are right, what they call "Datalog" without any qualification in academic papers is not.

zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
I believe you are referring to Datomic's pull syntax (https://docs.datomic.com/on-prem/query/pull.html). The way I see it is that this is an add-on to the query language, not a core part of it, since it applies to the output of the query only. This would be analogous to having a GraphQL server on top of Cozo's relational model. (In fact, in earlier versions of Cozo we did have something like the pull syntax, but we quickly decided that we do not want two distinct ways of querying data in the same database, nor do we want to create a GraphQL clone).

I personally find Datomic's way of doing things a bit too convoluted, as you need to

- learn how to define the schema

- learn how to transact data

- learn how to query data

- learn how to "pull" data after the query is done

and the steps are all very different.

zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
Differential-datalog is a cool project. I think the targeted use cases are different as compared to Cozo. The most important difference is that Cozo is focused on graphs, whereas differential-datalog is focused on incremental computation. These two goals are somewhat at odds with each other, as for queries with lots of joins (very common in graph computations), you can't know whether it's better to compute new results incrementally or to recompute everything until you actually run the query. Also, Cozo caters for the exploratory phase of data analysis (no need to define types/tables beforehand), whereas in differential-datalog everything must be explicit upfront.
zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
Re Datalog syntax: yes, the "composability" is the main reason that I decided to adopt it as the query language. This is also the reason why we made storing query results back into the database very easy (no pre-declaration of "tables" necessary) so that intermediate results can be materialized in the database at will and be used by multiple subsequent queries.
zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
Thanks for your interest in this!

It currently uses RocksDB as the storage engine. If your server has enough resources, I believe it can store TBs of data with no problem.

Running queries on datasets this big is a complicated story. Point lookups should be nearly instant, whereas running complicated graph algorithms on the whole dataset is (currently) out of the question, since all the rows a query touches must reside in memory. Also, the algorithmic complexity of some of the graph algorithms is too high for big data and there's nothing we can do about it. We aim to provide a smooth way for big data to be distilled layer by layer, but we are not there yet.

zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
I realized my mistake, as I said in the other comments. The main repo is now under LGPL. I'll see what I'll do with the bindings. Writing code is so much better than dealing with licenses!
zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
OP here. Nothing about the license is final yet since there are no outside contributors. I just changed the main repo to LGPL, not because what I believed in changed, but because it seems that I really misunderstood the licenses.
zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
It seems that I really did misunderstand the differences. It is now under LGPL. The repo still requires CLA for contribution for the moment until I am really sure.
zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
Thank you for your perspective.

Maybe I was confused about the case of using an executable vs linking against a library. Let me double-check with a few friends who understand copyright laws better than me. If everything checks out, the next release will be under LGPL.

About CLA: at the previous suggestion of a friend, the repo was locked with CLA requirement currently (even though nobody outside contributed yet). This will be lifted once the situation becomes clearer.

zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
Maybe, and maybe I need to consult a lawyer someday to get the facts straight. To tell you the truth my head hurts when I attempt to understand what these licenses say. Regardless, I intend this project to be true FOSS, the "finer detail" of which FOSS license it uses may change.
zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
No, my understanding is that if you don't make any changes to the Cozo code, you don't need to release anything to the public. If you do, and you cannot release your non-Cozo code, then you must dynamically link to the library (and release your changes to the Cozo code). The Python, NodeJS and Java/Clojure libraries all use dynamic linking.

There is no plan for any commercial license - this is a personal project at the moment. My hope is for this project to grow into a true FOSS database with wide contributions and no company controlling it. If a community forms and after I understand the consequences a little bit more, the license may change if the community decides that it is better for the long-term good of the project. For the moment though, it is staying AGPL.

zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
I find nothing wrong with the classical syntax, but there is a very practical, even stupid reason why the syntax is the way it is now. As you can see from the tutorial (https://nbviewer.org/github/cozodb/cozo-docs/blob/main/tutor...), you can run Cozo in Jupyter notebooks and mix it with Python code. This is the main way that I myself interact with Cozo. Since I don't fancy writing an unmaintainable mess of Jupyter frontend code that may become obsolete in a few years, CozoScript had better look like python enough so as not to completely baffle the Jupyter syntax highlighter. That's why the syntax for comments is `#`, not `//`. That's also why the syntax for stored relation is `*stored`, not `&stored` or `%stored`.

This is a hack from the beginning, but over time I grew to like the syntax quite a bit. And hopefully by being similar to Python or JS superficially, fewer confusion results for new users :)

zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
My use of "embedded" means that the whole database runs in the same process as your application. This is how SQLite works. Your application doesn't "connect" to an SQLite database in the usual sense. Your application simply contains SQLite as part of itself. Contrast this with Postgres, where you first need to start a Postgres server and then have your application talk to it.
zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
Yes of course.

Perhaps I should start by clarifying that I am talking about the number of queries the Cypher language can express, without any vendor-specific extensions, since my consideration was whether to use it as the query language for my own database. And Cypher is of course much more convenient to _type_ than SQL for expressing graph traversals - it was built for that.

With that understanding, any cypher pattern can be translated into a series of joins and projections in SQL, and any recursive query in cypher can be translated into a recursive CTE. Theoretically, SQL with recursive CTE is not Turing complete (unless you also add in window functions in recursive CTE, which I don't think any of the Cypher databases currently provide), whereas Datalog with function symbol is. Practically, you can easily write a shortest path query in pure Datalog without recourse to built-in algorithms (an example is shown in README), and at least in Cozo it executes essentially as a variant of Dijkstra's algorithm. I'm not sure I can do that in Cypher. I don't think it is doable.

zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
Transactions are described in the manual: https://cozodb.github.io/current/manual/stored.html#chaining....

Sorry about the docs being all over the place at the moment! My only excuse is that Cozo is very young. The documentation (and the implementation) still needs a lot of work!

zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
In short: yes, but not right now. See this issue: https://github.com/cozodb/cozo/issues/2. Also in this case you are not really using it as an embedded database anymore, which is our original motivation. We currently also provide a "cozoserver", but it is pretty primitive at the moment. "Big data" capabilities, when they arrive in Cozo, will probably go into the server instead of the embedded binaries.
zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
For the moment: you can have as much data as you want on disk as long as the RocksDB storage engine can handle it, which I believe is quite large. For any single query though, you want all the data you touch to fit in memory. The good news is that Rust is very efficient in using memory. This will be improved in future versions.

For the built-in graph algorithms, you are also limited by the algorithmic complexity, which for some of them is quite high (notably betweenness centrality). There is nothing the database can help in this case, though we may add some approximate algorithms with lower complexities later.

zh217··on A Sun-like star orbiting a black hole
It's actually quite normal for experimental and observational subjects, which is the case for this one. I believe the record is set by one with over 5000 authors - see https://www.nature.com/articles/nature.2015.17567 for details.
zh217··on Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite
I'm very happy that you like it!

The "teasers" section in the repo README contains a few examples. Or you could look at the tutorial (https://nbviewer.org/github/cozodb/cozo-docs/blob/main/tutor...), which contains all sorts of examples.

The Rust documentation on docs.rs could certainly be improved, will do that later!

zh217··on The Fall of Roam
Sqlite supports recursive queries. It is clumsy and not too easy to write, but it works.

For syncing, what I did was to implement a simple automatic backup logic. Then if I want to move to a new device, I just copy the backup I want, which is just a sqlite db file, and restore it. It is not ideal if you are moving from device to device constantly, but it has the benefit that you can compare backups to see what exactly you have changed (I even wrote a tool to build graph visualizations from the output of sqldiff).

zh217··on The Fall of Roam
I actually grew really frustrated with existing solutions for organising notes:

- Almost all solutions that attempt to take seriously the concept of knowledge graphs are Software-as-a-Service. Ouch. They say you should “build a second brain”. I certainly don’t fancy the fate of my second brain (whatever that may be) to depend on the commercial success of some company.

- All existing editors, even the web-based ones, have some hangover from file-based content management. Even with ability to cross-link to nodes inside a tree structure within files, the rigidity of file boundaries is still over-constraining, especially when one is trying to grapple with unfamiliar concepts during which the structure and links are constantly shifting.

- The maths-authoring abilities (i.e. facilities to edit LaTeX formulae) at best look like an afterthought. Writing anything more complicated than Maxwell’s equations is painful.

Out of desperation I decided to roll my own. I set out the following desiderata:

- It should take the concept of knowledge graph as far as possible. The foremost thing is to do away with files, and promote vertices (nodes) and links (relationships) to first-class status.

- While upholding the previous principle, it should make editing as natural as possible, preferably confirming to existing text-editing habits. The experience should definitely not be like creating nodes and linking them together inside graph databases: slow and confusing.

- It must work without an Internet connection. Or better: it should not make any attempt to connect to the Internet at all.

- It must not be too heavy-weight, and data should be easy to back up. This precludes any JVM-related stuff, docker images, or spinning up Postgres/Neo4j/Datomic/what-have-you just to write some notes on a local machine.

- It should cater for the needs of heavy maths editing.

Now I have been using my own app, powered by sqlite for storage engine and ProseMirror for frontend, for quite some time, and I am really satisfied (hard not to be satisfied by your own work though). It is becoming increasingly polished (full-text search anywhere, very fancy query, loops in the graph, etc.) but I hesitate to open-source it, because as far as I know maintaining open-source projects can be really time-consuming, especially when people actually start using them.

zh217··on Why Clojure?
I totally agree with the "missing core.async" part. Actually I miss them so much in python that I implemented the complete set of semantics in [aiochan](https://github.com/zh217/aiochan).

(disclaimer: I am the author of aiochan)

zh217··on Show HN: Aiochan – Golang-Style Concurrency for Python
Author here - thanks, would be really grateful for any feedbacks on the docs. The tutorial there is aimed at beginners who have never done any concurrency programming before, and it would be helpful if anything that is not thoroughly explained is pointed out.
zh217··on How Microsoft computer scientists and researchers are working to 'solve' cancer
http://xkcd.com/1736/
← PreviousPage 2 of 2