42 karma · joined March 6, 2024
It seems par2 is significantly faster with those options set than without, as in by an order of magnitude, it seems par2 struggles greatly with the large number of blocks that it sets by default. Thank you for telling me.
Mind you, bursts greater than a certain size, around 8000 bytes for the defaults, are a game of probability in what damage they will do. In most cases it'll work like you did for you, but in some unlucky offsets 10000 byte corruptions with default settings could do something like this.
1 byte corrupts fragment x of block n, 4096 bytes decimate fragment x of block n+1, 4096 bytes decimate fragment x of block n+2, 1807 bytes corrupts fragment x+1 of block n.
The defaults only have 1 parity fragment, so the fact two fragments were corrupted for block n means it can't be faithfully reconstructed. In the average case you should be fine, but it is something I should probably communicate more effectively in the README (I only hint at it with the 8K per 192K comment).
Yes, as you need to read the entire set of interleaved blocks to get each respective block, so to keep memory consumption low, we don't want to interleave too many blocks. I could increase the default to something higher though.
Regarding the tape problem, I was being a bit daft in my response, as if you lose a chunk of tape, you also lose some indeterminate number of data with it, which my format currently isn't capable of recovering from. I'll try to fix that to some degree in the next version, at the cost of not guaranteeing backwards compatibility as I feel it falls under that major problem comment I made in the v0.1 release.
I can't reproduce this. These are the commands I used, with it freshly compiled on a ubuntu docker container, both the v0.1 release and the master tree.
./bef -c -i Makefile -o Makefile.bef
dd if=/dev/zero of=Makefile.bef bs=1 oseek=300 count=1
./bef -d -i Makefile.bef -o Makefile2
cmp Makefile Makefile2 || echo "failed!"
edit: oh, I see, you 'removed a character'. Depending on what character you removed or corrupted from the output, you could've either hit the issue described above with inserting noise, but this time removing information, or you caused corruption in the header by either corrupting the magic number, hash type, or hash itself. The command line utility automatically truncates the output before calling the deconstruct function. The header is sadly the biggest single point of failure in the tool/format, which is why I introduced the --raw flag for those who don't want it.
In terms of controlling how many blocks and their respective fragments are interleaved, that can be controlled by the -l argument in the command. I'll paste the info I have in the manpage here.
>The number of blocks(specifically their fragments) to interleave. The default number is 3, as it provides protection for a bad burst to corrupt both the block in front of and behind it.
In general, you can approximate the burst size ratio needed to destroy a set of interleaved blocks beyond repair via this equation (this may not be fully accurate as its napkin math)
let n = number of blocks to interleave, B = number of bytes per fragment, m = number of parity fragments, k = number of required fragments to rebuild a block.
((n - 1) * m * B + n - 1) / (n * (k + m) * B)
this is because a given corruption could traverse a whole fragment and then leech into another, taking some other fragments with it with the most unlucky of bursts. When you remove the B and take the limit as n goes to infinity, it approaches the ratio of m/(k+m), or in other words as you interleave more and more blocks, you get a maximal burst size closer and closer to the size of your total number of parity fragments.
Also, the header specifies exactly the size of each fragment, we don't depend on the fragments themselves to tell us their size. In fact, the only metadata a fragment has is its hash and the number of padded bytes for the whole interleaved set of fragments, the format is (almost, ignoring padded bytes) completely described from the header as a design principle. That's why a whole fragment can be turned to garbage and we don't care, just skipping ahead to the next fragment.
edit: thinking on it, I could add a mode of behavior such that it recovers as much as possible if a specific block is corrupted, rather than the situation right now where it quits streaming after it finds a corrupted block.
bef -c --default share -i input -o output, bef -c --default dvd -i input -o output, bef -c --default tape -i input -o output, etc.
It seems like a good idea and wouldn't exactly be hard to implement.
tar c dir | zstd | gpg -e | bef -c -o backup.tar.zst.gpg.bef
and then to get back that file with the terribly long filename
bef -d -i backup.tar.zst.gpg.bef | unzstd | gpg -d | tar x
This is a really important property in situations where there can be big giant bursts of errors, because you can still reconstruct the data regardless. IIRC, CDs/DVDs/BDs all use two concatenated Reed Solomon (a type of erasure coding) coded symbols that are then interleaved with each other, which provides the disk protection against things like accidental scratches.