Hacker News Reader

Comment by chromedev | Hacker News Reader

Have you ever used a flat file database? I envision the perfect database having a file system API that automatically sorts and can be edited using standard files, but stores data using like JSON or YAML. That way it can be versioned and queried using the standard file system tools, or those that implement the Unix Philosophy.

View on HN

I tried this for a small project once about 10 years ago, using svn as version control and putting a GUI around it. It was clunky, with no good way of doing uniqueness or foreign key constraints, and slowed way down after ~1000 records. I ended up moving to a regular database and implemented the versioning logic manually, which was pretty straightforward.

At first this irked be because the final solution didn't retain the full power of a VCS. No merges, no branches, etc. But what I finally came to terms with was that it didn't need to (YAGNI): it just needed historical records and that was pretty easy. It ended up being much cleaner and faster that way.

I think that you'd have to also implement indexes to make that work, just like SQL databases do. Even with SQL, I ran into plenty of cases that indexes were necessary for decent speed. Flatfiles would just amplify that.

And, if you're trying to use the VCS to do merges, the indexes will constantly get messed up, so they'd have to be reindexed after every merge to be sure they were legit, and it still wouldn't ensure uniqueness... Ouch.

I'm building something like that as a side project and might have something to share in a week or two.

Reply on news.ycombinator.com