Comment by tzs | Hacker News Reader

Comment by tzs | Hacker News Reader

Parent Full thread

I was about to use Backblaze Personal and then noticed that. Was then about to use Arq + Backblaze B2 when I realized that (1) Arq supports OneDrive, (2) I have a TB of OneDrive as part of my Microsoft 365 subscription, and (3) I only use cloud storage for file transfer between mobile devices and desktop, so my OneDrive space was almost all unused, so went with Arq + OneDrive. 3 years of weekly backups later and I've still only used about half of my 1 TB.

The only thing I'm not happy about with Arq is that a "verify" downloads all the backup data to checksum it. That takes 3 or 4 days on my connection.

I thought I read that many cloud storage provider APIs provide a way to ask the server for a checksum of a stored blob. I'd have expected Arq to make use of that, but maybe it is not reliable (the server might just report what the checksum is supposed to be, not actually read the data and checksum it?).

Arq documents their storage format. I wonder if it would be possible to use a VM on Azure to access my OneDrive storage and do the checksumming on the VM?

Yes, online storage services do store a checksum of the files. They can use that internally to periodically scrub their stored data. For example, if S3 has stored 3 copies of your file, they can use the hash to distinguish good and bad copies, and replace bad copies. For erasure-coded services like Backblaze B2, they can reconstruct a file using any 17 of 20 erasure segments, so with a known checksum they can figure out which segment is bad, delete it, and use the other segments to generate a replacement. I was told by Backblaze that they do periodically scrub B2 files.

HashBackup (I'm the author) has a lot of ways to verify backups. For B2 uploads, it generates a SHA1 hash before sending the data, stores that in a local database, sends the data with the hash, B2 generates an independent hash of the data, and compares it to the one HB sent. If there is a mismatch, an upload error occurs and the upload is retried.

On download, B2 sends the data and SHA1 hash (I'm assuming it verifies the hash during erasure reconstruction also), HB generates a SHA1 hash for the received data, verifies it against the SHA1 B2 sent, and verifies it against the SHA1 stored when the file was uploaded.

After the upload, files can be verified by:

- downloading the complete file as above, then verifying every block in the file - getting the file size and SHA1 hash from B2 and verifying those against the local database (no data download) - downloading N random samples from each file and verifying the block-level SHA1 for each sample - downloading a small percentage of the files every day to verify the backup over time (this can be done with whole files or random samples)

If a problem is detected and you have other copies of the backup on another storage service or locally, HashBackup's selftest will fetch all copies of a file to reconstruct the original backup data to correct any errors, then upload the corrected file to all destinations.

Thanks a lot for your work on HashBackup. I can't recommend it enough. It has been great and working for me for years, no data loss and multiple restores. Top notch documentation too, just a bit sad that it isn't open source so that I could contribute. But the quality is there and it has been really solid for me.

Arq backup to OneDrive used to be very slow since IIRC they were using an old version of the OneDrive API with pretty ridiculous latencies. How is it these days?

Arq 7 uses the Microsoft Graph API, the one we're supposed to use now.

Thanks for the reply!

The "back up and validate" function only downloads all the data if you're using a storage location that doesn't provide checksums (SFTP or NAS/folder). So, it shouldn't be downloading very much if you're using it with OneDrive. Please email support@arqbackup.com so we can follow up on this. Thanks.

I don't have automatic updates enabled and it has been a while since I manually updated. I just updated and the "back up and validate" menu item is no longer there for my OneDrive backup. It is still there for my backup to a local removable drive.

I see from this Reddit thread [1] that it was removed for storage locations that don't need it a few months ago, saying

> Back Up and Validate is a waste of time and money for any storage location that provides checksums. That's why we removed it. It only causes pain and cost.

My last "back up and validate" to OneDrive was a couple months before you removed it, so it looks like I was just hitting the "waste of time and money" case that prompted its removal.

From the Reddit thread, it looks like the way to do a check that everything is OK on both ends when using a storage provider that provides checksums is to clear the cache and to modify the backup plan (just editing it and saving is sufficient), and then do a backup. The cleared cache makes it get the checksums for everything in the backup and the modified backup plan makes it rescan everything locally. Is that correct?

[1] https://www.reddit.com/r/Arqbackup/comments/n87vuw/missing_b...

There's no need to edit the backup plan. As I wrote in the reddit thread, clear the cache and do a backup. Sorry it's not more clear.

Reply on news.ycombinator.com