Hacker News Reader

Comment by eska | Hacker News Reader

HDDs also have a spinning medium and a read head , so the optimization is similar to optical media like CDs.

Let’s say you have UI textures that you always need, common player models and textures, the battle music, but world geometry and monsters change per stage. Create an archive file (pak, wad, …) for each stage, duplicating UI, player and battle music assets into each archive. This makes it so that you fully utilize HDD pages (some small config file won’t fill 4kb filesystem pages or even the smaller disk sectors). All the data of one stage will be read into disk cache in one fell swoop as well.

On optical media like CDs one would even put some data closer to the middle or on the outer edge of the disc because the reading speed is different due to the linear velocity.

This is an optimization for bandwidth at the cost of size (which often wasn’t a problem because the medium wasn’t filled anyway)

View on HN

> HDDs also have a spinning medium and a read head , so the optimization is similar to optical media like CDs.

HDDs also have to real with fragmentation, I wonder what the odds that you get to write 150 GBs (and then regular updates in the 30GB range) without breaking it into fragments...

64MB...

Microsoft has a paper somewhere that shows IO speed starts to drop when fragments of files get below 64MB. So you can split that file up into a few thousand pieces without much performance loss at all.

The game installer can't control the layout on an HDD without doing some very questionable things like defragging and moving existing user files around the disk. It probably _could_ but the risk of irrecoverable user data loss or accidentally corrupting a boot partition via a bug would make it completely not worth it.

Even if you pack those, there's no guarantee they don't get fragmented by the filesystem.

CDs are different not because of media, but because of who owns the storage media layout.

It's less about ensuring perfect layout as it is about avoiding almost guaranteed terrible layout. Unless your filesystem is fully fragmented already it won't intentionally shuffle and split big files without a good reason.

Single large file is still more likely to be mostly sequential compared to 10000 tiny files. With large amount of individual files the file system is more likely to opportunistically use the small files for filling previously left holes. Individual files more or less guarantee that you will have to do multiple syscalls per each file and to open and read it, also potentially more amount of indirection and jumping around on the OS side to read the metadata of each individual file. Individual files also increases chance of accidentally introducing random seeks due to mismatch between the order updater writes files, the way file system orders things and the order in which level description files list and reads files.

I don't disagree that large files are better, or at least simpler. I do think gaming drives are more likely than most to be fragmented (large file sizes, frequent updates plus uninstalling and re-installing). A single large file should make the next read predictable and easy to buffer.

I am a little curious about the performance of reading several small files concurrently versus reading a large file linearly. I could see small files performing better with concurrent reads if they can be spread around the disk and the IO scheduler is clever enough that the disk is reading through nearly the whole rotation. If the disk is fragmented, the small files should theoretically be striped over basically the entire disk.

Reply on news.ycombinator.com