PyFileFixity: Helping long term storage of data via redundant ECCs
pypi.org
pypi.org
> Parchive (PAR1, PAR2, MultiPar): well known error correction file generator. The big advantage of Parchives is that an ecc block depends on multiple files: this allows to completely reconstruct a missing file from scratch using files that are still available. Works good for most people, but most available Parchive generators are not satisfiable for me because 1- they do not allow to generate an ecc for a directory tree recursively (except MultiPar, and even if it is allowed in the PAR2 specs), 2- they can be very slow to generate (even with multiprocessor extensions, because the galois field is over 2^16 instead of 2^8, which is very costly), 3- the spec is not very resilient to errors and tampering over the ecc file, as it assumes the ecc file won’t be corrupted (I also tested, it’s still a bit resilient, but it could be a lot more with some tweaking of the spec), 4- it doesn’t allow for partial recovery (recovering blocks that we can and pass the others that are unrecoverable): with PAR2, a file can be restored fully or it cannot be at all.
I had, from memory here, md5sums of data, including in blocks of 64k or so; if the media failed, and I needed that data, I could picture a way to modify par2 software and piece together chunks from the par2 blocks.
It's an eccentric user base to begin with, of course. Most archivists and ordinary users just create multiple backups, and the file integrity concerns aside from file sharing and archiving are best addressed at the filesystem level.
It would obviously be nice if PAR were faster, for example.
An archival programming effort which itself has fallen behind? I hope not, these sorts of things are pretty important.
Is it comparable?
Assuming they still use it.
https://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/ha...
Seems to have become unmaintained though, which is worrying.
Instead, people seem to have invested a lot of effort in going the other direction; let the application have less control and let many magic layers in the middle figure out the details. We have had filesystems for decades, which pretend to map names to a list of durable block numbers. Of course, the blocks are not so durable (nor is the database), so people lose their data all the time and act like it's normal.
People have a solution to that problem, though, make the block devices more durable! We can just add software or hardware RAID on top of a bunch of disks, and now suddenly what the filesystem thinks is just one block is now 3 blocks sharded across 3 disks. Some magical curator process then tells me when disks have failed (as though it can detect that with 100% reliability), and I pop in a new drive and magically everything has been repaired. Of course, in reality, the entire disk doesn't just disappear all at once; some blocks just return data with a flipped bit, or all 0s, or an occasional error. My experience in running RAID-1 arrays under Linux (for my workstation) is that I notice some errors in the logs and have to manually figure out which disk contains the correct block (this is why I always use 3 disks) and then dd the block from the good device to the degraded device. By the time the automatic curator shows up to detect and repair this problem for me... all three disks have been RMA'd. It is awful and I'd be surprised to hear a story where someone says "one of my disks failed, I was notified in a timely manner, and my array was repaired when I inserted a new disk". It just doesn't track enough data to actually work, because it ignores the important of error-correction codes. (Don't get me started on RAID-5. That has some parity, but not Reed-Solomon codes, and the parity just makes it unlikely that you could ever manually get the array out of a bad state.)
But even RAID isn't reliable enough, and people don't use it, so of course the disk pretends it is more durable than it actually is. Spinning rust disks have some spare sectors. If their internal heuristics that you cannot tune or read the source code for think something is wrong, it will store some data saying "now block 42 is actually stored at block 2389473" and my RAID controller, filesystem, and application never have to know! Hooray! SSDs are even more egregious; they amplify writes, they garbage collect, they compute proper R-S codes over the data, and have a multi-core CPU running god-knows-what code handling it all.
I think it is all crazy. In my dream world, an SSD would just be a bunch of flash-chips that are not very durable due to the nature of how TLC NAND flash works. Then my application can decide "this is an important photograph, so I'm going to shard it across 8 chips on two different controllers and use error-correction codes that can correct 100% of bit errors". But, that does not exist.
It all makes me mad because when I worked at Google I actually had control, at the application level, over all this. There was the underlying disk API that you could use directly that just stored a block, with the durability of a datacenter disk drive. Then on top of that were filesystems and databases where you had to pick the durability for every operation; you could pick between straight redundancy (write to 3 disks) or whatever level of Reed-Solomon correction you wanted (all these are tradeoffs between read speed, write speed, and actual durability).
Meanwhile in the real world, people build crazy things that still lose their data. ZFS loses data. Cloud block stores lose data. Consumer databases lose data. I know photographers and videographers that have lost all their work because of disk failure... they had multiple disks, but their applications did not uses the disks efficiently in a way that could detect and correct errors. So you come back a year later, and opening your file just says "sorry, error!" and you accept your loss and move on. Someday, I hope this can stop. I never want to watch a YouTube video like this again: https://www.youtube.com/watch?v=gSrnXgAmK8k (LTT's Windows-based fileserver with hardware RAID fails randomly and cannot be fixed; all their work gone)
And that solves the problem in the sense of "I have some form of redundancy", but not "I am confident that it's durable", and definitely not "I am actually curating the data at archival quality".
But when I investigated using PAR2 I had another thought, which is that, in a sense, this is a quality-of-service type of issue, and those are always subject to variable standards. And part of my problem is like what you allude to - I can't easily specify how important the data is. If only I could just say, "I only somewhat care about these commercial multi-gigabyte sample libraries, but I care deeply about my password database, so please do the right thing with each." And that's something that - whether it's surfaced to the app or to the user - it really needs a much greater degree of coordination to pull off. As it is, there's a lot of redundant guesswork taking place.
It’s still not sufficient to backup to a single drive since the device may be lost or destroyed. And once you are storing in the cloud you have all the benefits of distributed redundancy and multi-regional support.
The cloud is not made of magic. There are some things that work well, like immutable object stores that are okay with eventual consistency (S3 as an example), but these stores do not work well for things like databases, or logs, or editing video.