Looking forward to play with this!
The main thing I will be wondering now is how it will scale to really large datasets. Any input on that?
Looking forward to play with this!
The main thing I will be wondering now is how it will scale to really large datasets. Any input on that?
It currently uses RocksDB as the storage engine. If your server has enough resources, I believe it can store TBs of data with no problem.
Running queries on datasets this big is a complicated story. Point lookups should be nearly instant, whereas running complicated graph algorithms on the whole dataset is (currently) out of the question, since all the rows a query touches must reside in memory. Also, the algorithmic complexity of some of the graph algorithms is too high for big data and there's nothing we can do about it. We aim to provide a smooth way for big data to be distilled layer by layer, but we are not there yet.
what if you had not so many nodes but each node had a lot of data would that improve it? Probably not but just normally I think of the number of nodes in your graph as the problem.
For your second question, say you have a relation with lots of fields, one of them particularly large. As long as you don't use that field in your query, it will not impact memory usage. The query may be slower though since the RocksDB storage engine needs to read more pages from disk, but the fields that are loaded by RocksDB but not needed will be promptly evicted from memory.
- The tutorial: https://nbviewer.org/github/cozodb/cozo-docs/blob/main/tutor...
- The language documentation: https://cozodb.github.io/current/manual/
- The pycozo library README for some examples on how to run this from inline python: https://github.com/cozodb/pycozo#readme