https://nest.pijul.com/pijul/sanakirja
Searching the web for blogs on this topic seems the best way to learn about it.
https://nest.pijul.com/pijul/sanakirja
Searching the web for blogs on this topic seems the best way to learn about it.
See for instance https://docs.rs/sanakirja-core/latest/sanakirja_core/btree/p...
Sanakirja even reached a point where bugs were found only in the last few functions without comments. Writing the comments to explain exactly what each line does in all cases (there are lots of cases everywhere) fixed everything.
It stores 4kb blobs, right? Does Pijul first parse the data (copying it to other allocations), or does it use the data as is? I mean, there are some libraries like cap'n'proto[0] and rkyv[1] that can directly use the file contents as an in-memory data structure, without a deserialization step (or rather, deserialization is simply a quick validation to check the data isn't corrupted), I was wondering if Pijul did anything like that.
And like, is this btree header [2] stored exactly like this on disk, and does Sanakirja exploits this to avoid further copying data?
(I guess there's a trouble with compression there: to decompress you really need to write in another buffer)
Also, is the I/O done with something that prevent userspace copies like mmap or io_uring, or does it eventually calls read() to copy the data to its own buffer?
I want to build something like Sanakirja, but with those features to avoid copying data, so I'm wondering if there's any overlap.
[0] https://github.com/capnproto/capnproto-rust
[1] https://github.com/rkyv/rkyv
[2] https://docs.rs/sanakirja-core/latest/sanakirja_core/btree/p...
The answer to all your other questions is yes.
I just saw it uses mmap. My other concern was that mmap handles I/O errors asynchronously through unix signals, and it isn't possible to pinpoint exactly which operation caused an error. Does this mean that a write error could cause data loss (other than what was written)? Or does Sanakirja guards against that?
The worst case scenario is that if the program doesn't know about a write error right away, ensuing read operations on the same location could handle corrupted data, which could lead to further erroneous operations, until the error is dispatched to the signal handler and hopefully the program exits.
All writes are done in atomic transactions.
I don't, the OS can flush anything it wants at any time.
Since you mentioned sanakirja, I'd like to ask: the obvious (to me) reason to build it, rather than use SQLite, is easy ABI compatibility with the rest of the project through Rust. Are there other architectural reasons it's particularly suited to pijul?
I ask because I already store as much as I can in SQLite databases, and am moving to patchsets and changesets for certain kinds of document-oriented data management. Storing pijul's artifacts in that same system would be a natural fit for my application, so I wonder how closely tied together the two codebases are on a practical level.
Yes, I originally wrote it because I wanted to fork tables without copying any byte. This is how channels (similar to Git branches) are implemented in Pijul.
> I wonder how closely tied together the two codebases are on a practical level.
Pijul can use a generic backend, although in practice only variants of Sanakirja have been implemented. Sanakirja is generic enough to handle different workflows, although large values aren't implemented yet (they aren't hard to implement, I just never needed them and nobody else did it).
As it happens SQLite added some of the machinery from fossil for no-copy tables and changesets as an optional extension. I mention this because it gives me hope that I might cobble something efficient together in SQLite, I could offer a few reasons I'm glad Sanakirja exists aside from pijul and the advantage to your project of being able to tweak both sides of the system is compelling.
Congratulations again on the beta. Doing hard and basic work for years is to be commended.