Hacker News Reader

Comment by locknitpicker | Hacker News Reader

> "borg" has basically solved backups permanently. It's deduplicated, even across snapshots, compressed, and end-to-end encrypted.

Deduplication helps minimize space, but isn't it a major liability in backups? I mean, what happens when you try to restore your backups but a lone sector holding a file from way back in the past happens to not be recoverable? Doesn't it mean that no matter how frequent your backups are, your data is lost?

View on HN

In that case you are supposed to use your /other/ backup. Which you have.

Defeating the point of deduplication...

You don't have backups if you only have one "backup". Look up sysadmin's 3-2-1.

On a single drive, deduplicate all you want. But you need to have two drives, at least, anyway.

If you are storing your data in a filesystem like ZFS then the risk is lower since you'll know you have a bad sector long before you attempt to deduplicate. But I otherwise share your same concerns and I'm eager to hear from others how to mitigate this. I've lost a volume due to poor practice while encrypting so I'm understandably reluctant to overcomplicate my backup strategies in the name of space compression or privacy.

It doesn't really hurt in practice because it's only one part of the full backup procedure. Deduplicate to save space; re-duplicate the (smaller) backups to separate media for redundancy; scrub regularly for bit rot and rotate your media. A proper backup system requires all but the first of those anyway.

This is why Proxmox Backup Server has scheduled "verify" tasks which look at the sha256 of every stored hunk to make sure it hasn't rotted on disk. (If it has, it makes sure to re-acquire that data on the next backup - hopefully it's still available!)

the backups in borg and restic are not classical copies, each snapshot is not independent, it is a collection of hashed blocks. In a sense, time machine and hard links do the same: they reference the same bits across snapshots, saving space. for the bit rot / faulty sector or failed drive, you use redundancy with RAID or similar in your file system. if all else fails and your backup system burns in a fire, then you reach for your other backup. deduplication and redundancy solve different problems.

This is bordering on paranoia. If the bad sector contains a critical part of the filesystem, you're going to lose everything anyway.

Do modern disks even have physical "sectors" anymore? Isn't it all virtual?

Without dedup you're just going to backup less stuff, which is far worse.

> This is bordering on paranoia.

What? You actually think that the mere idea of a backup getting corrupted is something that is "bordering on paranoia"? Have you ever heard of redundancy or even RAID?

> If the bad sector contains a critical part of the filesystem, you're going to lose everything anyway.

Do you honestly failed to understand that the problem is actually the "lose everything" part?

Reply on news.ycombinator.com