Hacker News Reader

I noticed when a discussion like this comes up people separate themselves into two camps that mock each other's assumptions while both talking about the same thing. It would be nice to talk more concretely with benchmark numbers.

1. What about a database saves you disk space? Whatever it is, the filesystem can do the same thing. Perhaps you had a preconceived notion of a specific filesystem and OS in mind when you wrote this.

2. Your database MMAPs/caches pages with its custom page buffer too, and that wastes ram with small records.

3. Reading/writing data from a database requires syscalls too. An OS can cache a lot in memory and minimize disk writes. Arguably the database is duplicating the caching logic of the OS and slows things down.

4. Filesystems have atomic transactions too. You can use the flock call to lock files, and rename is atomic. It's a rather befitting hack for HackerNews which "atomically" renames a file to update it.

5, 6 and 7: You can get distribution, replication, fault-tolerance, availability, and snapshots for free out of a filesystem like GlusterFS too. (Also note SQL is crap; it's a bad serialization of a data structure.)

I wonder if the real reason we have better databases than better filesystems is people didn't want to program for the kernel, which is where filesystems had typically ran.

People not grasping you can store data on a filesystem is probably one of the most valuable things a new programming language would have going for it. It would be a staggering reduction in complexity if a language made data manipulation easy, perhaps at the dismay of programmers stripped of the amusing complexity of using this separate thing called a database.

Could you clarify on 2? I mean, there's of course a per-row overhead on a DB, but many rows get put into the same page - so I'm not sure how that's remotely as wasteful as a hypothetical filesystem that uses a minimum of a page-per-file. Of course, that's not a requirement for FS behaviour - just historically a lot of them have worked that way.

> Reading/writing data from a database requires syscalls too

Not for stuff that's in the DB's page cache.

> Arguably the database is duplicating the caching logic of the OS and slows things down.

That's a pretty biased assessment. A 'real' database like Postgres will use calls that bypass the OS page cache, so there's no duplication there. It also understands its usage patterns better, so it can do generally better than a general-purpose cache.

> and rename is atomic

Assuming your file system doesn't atomically give you a truncated file post-system-crash :-).

> (Also note SQL is crap; it's a bad serialization of a data structure.)

Agreed that SQL is not the best language, but is this an argument against set-based query languages, or just SQL specifically?

Databases make some things that are really hard less hard. Managing concurrent read/writes over sets of files (say) is currently difficult. Sure, your FS could have deadlock detection, MVCC and so on built into it, but that would be turning the file system into even more of a database system than it already is. Database systems are complicated because people often want to do complicated things.

A way to execute set-based queries is a major point you bring up here. It's also the most baffling. When tens of filesystems with sophisticated features completely omit what is one of the most common features used by database users, presumably it's not because filesystem designers are not capable of implementing it but of some more mundane reason.

> Not for stuff that's in the DB's page cache.

The database has to log what was written. It does use syscalls to write the write-ahead log. Agreed it can save read syscalls. Then again one process sending a query to another process to read something generates even more syscalls than having everything (app+db) all in the same process.

You are right about 2. I'm playing devils advocate here jumping around assumptions. I could ask why you assume an OS will use a page per file, you can say Linux does this, I can say why is your OS so heavy and point to Exokernel, you can say that's side-stepping the issue. I could say user-space filesystem having its own page cache; in the end we are talking about the same thing. In the end: the less layers the better.

When I look at the kernel I see something like the Berlin wall. A barrier that was there since you were born and which you never asked for. A kernel being hard to hack and monolithic is bound to push developers away, but it won't stop developers from building what they want in the end.

"It would be a staggering reduction in complexity if a language made data manipulation easy,"

So, you mean, like a query language?

"perhaps at the dismay of programmers stripped of the amusing complexity of using this separate thing called a database."

Or facing the prospect of learning another database system, and unsure what the gain is.

"future devs will have to learn your crazy system rather than leveraging all the existing SQL knowledge they currently have."

Super important point that sometimes trumps everything else when considering any new technology that people often forget. Some of the first servers that I purchased were SGI running Irix. Turns out that the person who advised that didn't consider the fact that while the Sun's weren't "as good" he also didn't consider that there were so many more packages available on Sun's as well as sysadmin's as well as information (at that time) on the net.

What I've found over time is that unless there is a compelling reason to do otherwise you go with the ubiquitous solution to a problem.

Back in the day: Or the solution that there was an O'Reilly book on or a row of books on the bookshelf explaining (when that is essentially the knowledge base).

I think your points are valid only for very different data scenarios.

1-2 scenario: many many small files.

3: continuously accessing your data

4: concurrent connections

5: distributed application

6: never-stopping application

7: concurrent writes to mutable data

You may need a DB, if you want several of these to happen. This might be true for many applications, but for many applications this is equally not true.

Most often than youd' expect, using a DB makes understanding an application _harder_ because the data in the application tends to get structured as in the database: only ids, ints, strings and maybe date, no graphs or algebraic data types such as sum types, optional types, etc.

That's a fair point, for many scenarios this stuff doesn't matter, like a video-game. However, most apps fall into either the "serialize a bunch of crap to disk" category, or the "multi-user data changes all the time category". Nearly everything in the second category works best with a proper database.

There are plenty of databases with support for more complex types, and in any case, it's not like the filesystem understands those either, so if you want to store something to disk, you'll need some kind of translation/serialization anyway.

I'm interested in the databases that support more complex types. Pointers?

Nothing very specific, I was specifically thinking about graph databases, since you mentioned them. There's also domain-specific datatypes, such as those implemented by PostGIS.

For graphs, try a graph database like Neo4j, OrientDB, Titan, InfiniteGraph or Dex.

Modern file systems can store several files' "tails" in one block, therefore they don't round file sizes to block size.

Interesting. How modern are we talking exactly? More ext2-modern, XFS-modern or ZFS-modern?

ext4 http://kernelnewbies.org/Linux_3.8#head-372b38979138cf2006bd...

That link talks about putting the entirety of very small files into their inodes. I don't see anything about combining the tails of multiple files in a single block.

That's true, but OP talked about storing '100 byte records' into a single file. For large files (x times block size), wasting a block for the tail is usually negligible.

I believe ReiserFS did that.

Thanks!

1) 100% yes 2) yup 3) Frequently read/written files will be stored in ram 4) This still affects DBs, as they are still a layer above the FS. 5) GPFS, GFS2, ceph, cleversafe, tahoe-lafs, all make this go away. 6) ZFS has snapshots. Snapshots are amazing. Granted they are not nice as SQL dump. 7) If you're doing this with files, you're in trouble.

Having said that, your last point is 100% valid. Don't reinvent the wheel. SQL is good for most things. Even SQLite is super awesome for slightly busy websites(depending on how you use it).

Agreed, but remember, syscall overhead is syscall overhead, cached in ram or not.