Ash HN: What if we use file system as database?

I'm going back to the file system as a source of truth with static site generators.

The main reason is that I can use a variety of tools to view and manipulate files.

But other software can also mess with your "database", including the operating system itself. For example, Syncthing was breaking my static site by changing the unicode normalisation of file names. MacOS being case insensitive brought issues. Illegal filename characters brought issues.

This approach works well if you want to make data accessible to humans, but it's wildly inefficient if you expect machines to operate on that data, and comes with a few caveats.

Pick the right tool for the job.

For simple apps and app components, it's very convenient and manageable.

It becomes a problem when you: (1) scale up (2) have to deal with multiple relationships between objects. The "Database design" by Adrienne Watt posted in another comment covers the scale concerns well, but another scale problem she doesn't mention is hitting inode limit, at least if you're on a single machine. You can of course use a distributed filesystem as database, but at that point, you might want to use a database proper.

This chapter from "Database design" by Adrienne Watt has a very good explanation why this approach didn't stick: https://opentextbc.ca/dbdesign01/chapter/chapter-1-before-th...

Why not, is because you want ACID guarantees (ie, what if two people are editing a file at once), ability to scale, or a query language.

I've been using a file based database at work for years. Data is stored in 1024 byte "blocks" with each file containing 1000 blocks. Each block refers to a single record and the data is delimited by the positions within those 1024 bytes (e.g. date/time created might be at position 38 in the block and run for 14 bytes so we reference it by its "token" of 380014). Each file will be the first 4 characters of the 8 character long primary IDs of each block. E.g. file 1020.dat would contain records 10200000 -> 10209999. We then have all kinds of other files used for indexing and data locations that are built by tools over those files as well as overflow data files when those 1024 bytes just weren't enough.

I should point out this is a legacy system and for very good reasons we moved to an actual database a long time ago.

In your example you're kind of mixing a DB within a DB (the index.json file is a separate database contained within the main database).

A better structure would be something like

    users/<user>/projects/<name>/data/... etc

Now your file system is just a NoSQL database. All that data you would dump into an index.json can be stored in the file tree.

That would actually work pretty well as long as you limit operations that are kind of meaningless and disable features like symlinks.

Mounting hierarchical data as a pseudo file system is actually pretty common. EG procfs, devfs, sysfs are all pseudo file system that present structured data to applications on Linux through the guise of a file tree.

Well you can use a database as the file system...That is what mobiles do. They use SQLite.

For many years, I built most projects on top of a relational DB.

Then NoSQL happened.

After NoSQL became less popular, I started defaulting to JSON files, if I thought I could make it work.

One thing that helps a file-based DB is to make sure you put something useful in the filename (and/or path). Such as a search tag and/or category or owner.

I suspect that the majority of programs are like the ones I write in that they don't have a ton of demands on them, relatively speaking. Not in terms of load or scope or anything else.

In most of these "small" applications, I think you can literally use almost _any_ database or file structure you want and end up with basically equivalent results.

For one, with a 1 file per row (entity) design, any larger query would require reading from potentially millions of files, which I imagine would be nightmarishly inefficient.

I have done this for simple web apps in the past.

And I have had programmers who used the resulting apps call me a liar and tell me it's not possible.

The file system is itself a multi-user indexed data store built right into the OS kernel. However, it obviously is not a relational DBMS.

So if your needs are limited and can be satisfied by a simple indexed data store, this can work remarkably well.

Even though some "software engineers" will tell you it can't because they have been trained differently.

Reminds me of WinFS, the Windows filesystem built on top of a relational database [0] that was supposed to ship with Longhorn (aka Windows Vista). Neat idea that never materialized.

[0] https://en.m.wikipedia.org/wiki/WinFS

It's fine as soon as the OS gives you "API", like searching, indexing, garbage colleciton programatically.

As long as you don't run into inode or file count limits, a filesystem is a valid way to structure data like this.

why index.json? you already have the file system's directory list of files