Designing File Formats (2005)
fadden.com
fadden.com
For the former, a static code that includes a CRLF does just fine and allows modifying the file without having to write code to recompute your annoying checksum.
For the latter, I find it questionable whether it's the file format's responsibility to prevent bit rot. What's it going to do anyway? Reject the file? Knowing whether the file is undamaged is good, but most of the time people will want it to do a best effort attempt at reading it anyway, and knowing whether integrity has been maintained is actually not that important for most files (if it is, it's usually a cryptographic feature and you'll not want to use CRC32 or MD5 as the author suggests). To recover from bit rot, you'll need a backup (or an error correction scheme but that seems out of scope of this article). If every file format has its own checksum, your backup solution can't check the files for integrity anyway, so it seems to me like it makes more sense to have checksumming be a filesystem feature.
I'd like it if checksums could just not be a thing in file headers anymore. The number of times it helped me is exactly zero, but I did run into issues when wanting to open an old file or modify a file. I would argue checksums prevent hacking in the HN sense of the word.
It also doesn’t help programs dealing with bad data. Apps still must handle invalid files gracefully, even if the checksum is good. Otherwise, hackers can create malicious invalid files with correct checksums.
Checksums also make small edits to a file painful, since every program must now know how to update the checksums. Imagine if text files had checksums - how could you possibly do simple regex replacements, e.g. using ‘sed’?
The page did mention doing it for the header, not the whole file
If possible, look into using a pre-existing notation like ASN.1 for this. No need to re-invent the wheel.
> The second-to-last byte is a Ctrl-Z, used on some systems as an end-of-file marker in text files. Not only does it detect improper text handling, it also stops you from getting a screen full of garbage if you "type" the file under MS-DOS.
Why does MS-DOS stop reading at an EOF market and not the true end of the file? I admit, I’m too young to have used DOS, but didn’t FAT(12/16) have file sizes?
> There are many, many file formats...
And one or more of them almost certainly fits the bill. It's 2019, "someone else has had this problem."
"Some words of advice on language design" comment on LtU by Frank Atanassow was mentioned yesterday in re: pg's Bel. http://lambda-the-ultimate.org/node/687#comment-18074
Everything in it applies to file formats as well, I think, eh?
(If you must make a new format specify a grammar and make it simple. https://en.wikipedia.org/wiki/Chomsky_hierarchy )
> It's 2019, "someone else has had this problem."
Sure, and that format only support 98% of what I need, and they're not interested in adding one new feature, or no longer available to maintain the format. Now I have a choice: write my own from scratch, or extend theirs in an unofficial way.
Or maybe they are willing and able to make a "1.1" version of their file format, and now 10 other programs that support the 1.0 file format won't support my files consistently, since they were never designed or tested for it.
As your LTU link suggests, in many cases:
> your solution belongs in a library, not a language
Sounds great, but what's a "library", in the world of file formats? It would be fantastic if every file format were extensible in all the ways that were useful to me, but they're not.
I don't quite understand the linked programming language design advice (i.e., if existence of SKI is a sufficient argument to discard languages based on cleaner models and fewer axioms, it seems that any language can be discarded on step 4 because there are Turing complete languages already), but I think a sensible course of action that would fit the "language versus library" analogy is to use an existing serialization format and/or data model, only defining its schema/ontology (if there is no suitable one). Similarly to separation of layers (particularly presentation and application ones, and perhaps down to transport layer, given the article's advice to include checksums) in network protocols.
There's no straightforward way, in any language/library/system I've used, to be able to say "take the lexer from file format XYZ, and I'm going to change the data model it produces" -- or vice versa. In theory I thought that's supposed to be what inheritance is about but nobody writes that way, and object-oriented programming isn't magic pixie dust you can sprinkle on any design to make it automatically reusable.
Apparently we've imagined slightly different things (and with the sibling comment too; perhaps I should have provided examples at once). What I had in mind is rather standard formats and markup languages, which are already used in practice (though not always): such as XML (SVG, OO XML, XHTML being based on it) and JSON; BER and others for standard encoding rules; RDF for a generic data model with various serialization options. That is, not taking a lexer from another format's implementation, but using a standard one. Likewise with integrity checks: many formats already rely on integrity being guaranteed by file systems and network protocols, and don't include checksums; a similar story with compression, and with encryption (both of which some formats would still redefine and include, but often they can be applied separately as well, or at least using an existing standard/algorithm/library).
HDF5 uses the same format, except of course with "HDF" instead of "PNG". I wonder if any other file formats have adopted this.