Hacker News Reader

Comment by tyingq | Hacker News Reader

"The database for a filesystem is an classic. WinFS, the filesystem that should have been a key feature of Longhorn/Windows Vista is based on a relational database."

I always wondered why Unix never added record-based files, in addition to stream-based files...like mainframes have. That would have simplified many things.

View on HN

> I always wondered why Unix never added record-based files, in addition to stream-based files...like mainframes have. That would have simplified many things.

You can implement record-based files (RAX, ISAM, etc) on top of stream-based files. The Unix philosophy was like Lego: small parts that do simple things that can be put together in combinations to build greater things. If you can build it from the basic blocks, it's not a basic block.

There are plenty of more advanced storage systems available for Unix. You don't need the OS vendor to supply The One.

The problem is triviality to implement, a similar problem occurs with TCP and UDP.

Lots of people implement half of TCP over UDP because they want packet based transmission but also care about losing packets.

Implementing packets on top of a stream is not as trivial as implementing a stream over packets.

Implementing streaming files over record-based files is easier, you just iterate over each record and append to the stream. Implementing records in stream-based files means you have to define a framing format, internal format, bookkeeping, etc.

Record-based files would be the proper lego building blocks.

The disk is a block device, i.e. it reads and writes data in blocks, the system just abstracts that into a stream interface.

Yeah and that is IMO the wrong abstraction. Blocks on disk should be organized into records, where multiple blocks can belong to a single or multiple (if using dedup or Copy-on-Write) records. Implementing a performant stream over that can be done in libc with minimal overhead.

Now you want a different block structure.

No? ZFS already organizes blocks on disks this way to some extend and Btrfs uses extends which include multiple blocks together and can be referenced multiple times.

> Implementing streaming files over record-based files is easier

What a performance-wise nightmare. Goodbye page cache and other zero copy optimizations.

Not really, the OS still has to abstract it onto the harddisk, this should be trivial to achieve without impacting page cache and zero copy optimizations if the on-disk metadata is seperated sufficiently. COW Filesystems already do it to a lesser degree and they're certainly fine.

What to do with clients who want a stream of records? Separated meta + payload allows fast byte streams. Combined meta + payload allows fast record streams. Structured implementation can't provide performance for both cases. But byte-oriented can. And we see this approach in modern and fast OSes.

Allow clients to choose which they receive?

May be I was unclear. If we introduce record abstraction for file ops then OS low-level implementation have to decide which way to store structured data. meta + payload or meta separated from payload. In latter case there is no way to provide fast record streams. Former case does not allow fast byte streams. Client decision to receive bytes or records doesn't magically zero copy reorder stored data.

Why would the later cause record streams to be slow? Snapshots in COW FS' aren't that terribly slow either, even if there is quite a few of them, which basically boils down to the same structure, really.

The problem with the TCP vs UDP debate is that there exists a middle ground that satisfies most people's requirements.

TCP offers a single reliable globally ordered stream.

UDP offers unreliable packets in arbitrary order.

What people usually want is to send whole packets(or multiple streams) in arbitrary order with configurable reliability.

TCP and UDP offer neither but UDP has the least restrictions which makes it the only protocol on which you can implement your custom protocol.

"You can implement record-based files (RAX, ISAM, etc) on top of stream-based files"

Sure. And we have, many times over. A base implementation that supported fixed and variable records, shared process mmap() access, etc...would have value. And perhaps the kernel being aware of it would have some benefit.

>If you can build it from the basic blocks, it's not a basic block.

The idea behind the author's post is that a directed acyclic graph is a basic block that is just one layer of abstraction above raw bytes. JSON, YAML, TOML, INI, .properties, environment variables are all just different ways to represent a graph and if it were possible to store the graph directly in the filesystem then one would not have to manually [0] serialize the graph to bytes and deserialize it back later.

[0] graph -> byte conversion is the job of the graph filesystem

Thank you for pointing that out, thats exactly one of my points.

> The Unix philosophy was like Lego: small parts that do simple things that can be put together in combinations to build greater things. If you can build it from the basic blocks, it's not a basic block.

And the normal programmer's logic is: "if it's not already integrated in the OS then I am not going to use it". So stream files it is, forever.

I think history has proven this. People are very risk-averse even if the risk is quite minor (example: using sqlite3 instead of the file system).

Reply on news.ycombinator.com