Hacker News Reader

ZIP is a hot mess but that particular feature is not a bad idea, it's just poorly implemented / designed.

The point was you could create a multi-disk zip (zip came out when we had 1.4meg floppy disks). If you zip up 10meg across 7 disks and you want to update the README of your 7 disks, by putting the central directory at the end you just ask the user to insert disk 7, read the central directory, append the new README, write a new central directory. With the directory at the start you'd have to re-write all 7 disks. With it at the end you only have to re-write disk 7 and only a small portion of the file.

The only bad part of that particular feature in the zip implementation is the scanning part. The variable sized underspecified "Zip comment" that you have to scan through should have either come before the central directory OR there should have been a length or pointer to directory after it so no scanning needed.

Also what would have been wrong with having a fixed size trailer/footer at the end of the file, pointing back to the (possibly newly appended) central directory, and even back to the previous trailer/footer if chaining is necessary/useful? No scanning necessary, then.

The PDF format does this correctly. PDF files end with:

    startxref
    12345
    %%EOF

where 12345 is the file offset of the TOC.

Many memories of PKZip asking me to insert the last disk of a set :)

Usually when copying a game...

And disk (total / 2 + 1) was the corrupted one, you'd only find it mid way through.

> You might think that ZIP files would have a header, or at least a fixed-width footer

ZIP files do have per file headers, which duplicate information in the directory, such as the file name. And malicious files have used this quirk to circumvent safeguards. See, e.g., WinRAR filename spoofing exploit: https://www.rapid7.com/db/modules/exploit/windows/fileformat...

Any time metadata is duplicated in a system (specifically, an interface or protocol), alarm bells go off for me. Simply duplicating metadata can inadvertently create an explosion of failure cases and implementation complexity. Conversely, avoiding duplication of data is an example of failure-proof design--if it's not duplicated, you don't have to worry about inconsistencies and reconciliation rules.

> Conversely, avoiding duplication of data is an example of failure-proof design--if it's not duplicated, you don't have to worry about inconsistencies and reconciliation rules.

I can't agree with this.

The reason you deal with inconsistencies and reconciliation rules is because, despite your efforts to the contrary, the world will give you inconsistent or incomplete copies of your data from time to time.

A sure-fire way to ensure that your data is lost is to only keep one copy of it. At some place in your storage stack, you want some level of redundancy. The general idea for an archive format is that it is somewhat self-synchronizing, and somewhat possible to repair. All media has a non-zero error rate. And then there are all of the various reasons (software, human error, disk full, etc.) that a file might get accidentally truncated (which destroys the central directory in a zip file).

You may not personally have to deal with the consequences of damaged media or similar types of failures, but for those that do, it's nice to have a data format where a stray error or two doesn't render the entire archive useless. If you are building on top of "reliable" storage systems (like cloud storage providers), you're just pushing the problem to lower levels in the stack... but there is less context at lower levels in the stack, which means recovery can't be very sophisticated.

This seems like a tension between the needs of genuine archival storage, and packaging of multi-file artifacts for web distribution. The latter use case dominates in the “archives” I actually encounter.

You can create multiple formats and expect your users to understand the differences and select the correct one, but the 99% use case is non-technical users--so 99% of us are going to choose the default option. A bit of redundancy (a short signature and copy of file metadata) is a clear win here, at least for popular formats like Zip. In order for a format to BE the default option for non-technical users, it has to be reasonably good for most use cases, rather than optimized for one specific use case.

If you are writing a package manager or backup program you might make a different choice, but how often do you find yourself writing a package manager?

> Any time metadata is duplicated in a system (specifically, an interface or protocol), alarm bells go off for me.

You might be very nervous to learn that most filesystems duplicate some metadata in one way or another. FAT, NTFS MFT, ext4 superblocks and group descriptors come to my mind. In some level you ought to have duplicates; it's a matter of where to place them. ZIP's choice made sense at the time of smaller and slower disks. (I do think that the modern standard ZIP should make some fields scrubbed to prevent compatibility problems, though.)

However if the metadata is duplicated you have the possibility of recovering data from a corrupted or incomplete archive.

> You might think that ZIP files would have a header, or at least a fixed-width footer, but they don't. Instead you're supposed to scan backward through the file looking for a magic number indicating you've found the central directory.

Yes. ZIP has no header, it has a footer.

What gets really messy is that the footer contains a variable length comment field, with the length stored before the field, so good look finding where the footer starts.

And here we are just talking a about a correctly generated file. This also opens up the door to all kinds of intentional abuse.

> It's amazing to me that this is the format that won the compression wars and stuck around to this day.

The header at the end actually helped it a lot. It allowed the self-unpacking archives to exist.

And allowed adding a file to a zip without needing to copy the whole zip to add header space at the start. A mere optimization today, but absolutely critical when working with floppies.

So can you technically "hide" data in a ZIP file as a form of steganography? Just add it to the zip file but mark it as deleted. Then any common zip utility will skip it, but a specialized one could extract that extra information. This would be easy to miss as the size of a zip file is always different than the sum of the size of the files it contains.

And of course googling zip file steganography reveals this is not a new idea: https://aroralakshay2014.medium.com/hide-it-lest-find-it-zip...

Oh yeah, that's the trick that lets you shove a zip into other file formats and have it be valid, right?

zip at least has a de-facto standard, which today is the "Info-ZIP" utility.

not really.

The zip spec says anything can come in before the "zip data". Info-Zip gives this example

    unzip unz552x3.exe unzipsfx.exe                 // extract the DOS SFX stub
    cat unzipsfx.exe yourzip.zip > yourDOSzip.exe  // create the SFX archive
    zip -A yourDOSzip.exe                          // fix up internal offsets

http://infozip.sourceforge.net/FAQ.html#unixSFX

In other words, append any data you want, then fix the pointers in the central directory.

There are tons of various zip utilities that will mess up depending on what you put in that self extractor including MacOS's built in finder based zip support.

PKWare, the keepers of the documentation for the ZIP format refuse to specify that you must read the end of central directory. They do this by claiming you can stream a zip file. But you can't stream a zip file if you have to read the central directory because data you may have already used, "the definition of streaming", maybe not be listed in the central directory in which case you shouldn't have used it.

In other words imagine each file in the zip is a compressed script and you are going to execute them. Imagine there are 2 scripts, the first one is "rm -rf /" and the second "ls /". The central directory only points to the 2nd script but if you stream you'll read the first script an execute it.

The ZIP spec claims you can stream but clearly you can't.

Zip also has a real bonefide standard: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT

That spec is seriously underspecified

https://games.greggman.com/game/zip-rant/

Seems kind of a shame to tuck this away in a comments section on a (mostly) unrelated article. I would suggest you submit it as a main article for HN :)