Hacker News Reader

Comment by mg | Hacker News Reader

Reading this again reminds me of the fact how beautifully git uses the file system as a database. Where everything is laid out nicely in directories and files.

Except for performance, is there any downside to this?

In other words: When you store data in an application that only reads and writes data occasionally, is it a good idea to use the git approach and store it in files?

View on HN

Performance is one problem, concurrency is another (you'll need another locking and logging system to make it concurrent-safe and atomic), it can also be unwieldy to move around, and it will be broken by dropbox-like apps that will mark individual files as conflicted (rather than your whole database).

Concurrency is not a big problem. The only concurrency issue with Git is that refs (and states like rebase/merge conflicts) are stored in "loose" files. This can be easily and elegantly solved like how jj does it, by putting the repo metadata into the object store too.

You can use a jj repo concurrently, e.g., over Dropbox with coworkers, and all it requires is a minor modification on top of the existing Git data model.

One major downside is that it becomes really hard to do transactions, especially across multiple files. If you store mostly immutable data though like git (where except the refs every object is immutable, mutating creates a new object) it can work nicely.

Hmm... is the mutability of data really enough to create a need for transactions?

For example here on HN (which afaik also stores the data in files) you can change a comment you wrote. But that type of mutability does not call for transactions, right?

Depends on the requirements, if you need concurrent access and things like durability in a crash you're going to be implementing something that looks a lot like a database.

Do files really prevent concurrent access?

And does git not need crash durability?

It's called flat file storage and it has a number of advantages and disadvantages. I prefer it because it is more robust and performs well, even for large amounts of data, as long as your filesystem is reasonably fast and reliable.

Think maildir vs. mbox/PST/etc. for message storage. I stopped counting the number of times that I have seen Outlook mangle its message database and require a rebuild.

Generally it is not so popular, in part because of the OSes like Windows and macOS which have somewhat lacking filesystem implementations. Git also has performance issues with large repositories on Windows, which need to be worked around by various methods.

Transactions are another limitation as mentioned by the other reply (they are possible to implement on top of "normal" filesystems, but not in a portable way).

Reply on news.ycombinator.com