Why hasn’t someone made sqlitefs yet?
> The performance difference arises (we believe) because when working from an SQLite database, the open() and close() system calls are invoked only once, whereas open() and close() are invoked once for each blob when using blobs stored in individual files.
https://github.com/narumatt/sqlitefs
And it seems quite interesting:
"sqlite-fs allows Linux and MacOS to mount a sqlite database file as a normal filesystem."
"If a database file name isn't specified, sqlite-fs use in-memory-db instead of a file. All data will be deleted when the filesystem is closed."
I wish I could find that comment, because my explanation doesn't do it justice. Very interesting idea, someone's probably going to explain how it's already been tried in some old IBM database a long time ago and failed due to whatever reason.
I still think it should be tried with newer technologies though, sounds like a very interesting idea.
The original article effectively argues the opposite: if your use case matches a database, then that will be way faster. Because the filesystem is both fully general, multi-process and multi-user, it's got to be pessimistic about its concurrency.
This is why e.g. games distribute their assets as massive blobs which are effectively filesystems - better, more predictable seek performance. Starting from the Doom WAD onwards.
For an example of databases that use the file system, both the mbox and maildir systems for email probably count?
It's now "somewhat possible" in that modern filesystem are overall mostly less broken about handling large number of small (or at least moderately small) files than they used to be.
But databases are still far more optimized for handling small pieces of data in the ways we want to handle data we put into databases, which typically also includes a need to index etc.
They basically mmap’ed the database file and argued that OS cache should do its job. Which makes sense but I guess it did not perform as well as any fune tuned caching mechanism.
What do you expect the value proposition of something loosely described as a sqlitefs to be?
One of the main selling points of SQLite is that you can statically link it into a binary and no one needs to maintain anything between the OS and the client application. I'm talking about things like versioning.
What benefit would there be to replace a library with a full blown file system?
I guess querying by PK has some similarities but it is not as unstructured and random as a seek.
Also side effects such as sparse files do not mean much from a database interface standpoint.
https://www.sqlite.org/c3ref/blob_open.html
https://www.sqlite.org/c3ref/blob_read.html
I've not seen performance numbers for those. Could make for an interesting micro-benchmark.
Using SQLite let's you tailor your data access patterns in a much more rigorous way and side step the POSIX tarpit.
It seems to me that all the extra indirection from using FUSE would lead to more than a 35% performance hit.
Statically linking an sqlite into a kernel module and providing it with filesystem access seems like something non trivial to me.
Putting a relational database in the OS kernel is an interesting violation of standard layering. Obviously has the potential to unleash a lot of issues, but also could possibly enable novel features.
It ships with an example VFS which shows you how to do this: https://www.sqlite.org/src/doc/trunk/src/test_onefile.c