Hacker News Reader

Comment by Cthulhu_ | Hacker News Reader

Of note is that since Backblaze is a backup company, their hard drives will probably not see a lot of churn / activity - write once, wait until something happens. Maybe refresh the data every once in a while.

I'd like to see similar tests from companies like Amazon and Dropbox (although the latter probably uses the former), in various use cases.

View on HN

Wouldnt they see the same amount of churn a typical user would see?

I do not use them but they say "That’s why Backblaze backs up your data automatically and constantly looks for new and changed files to backup. Install Backblaze and never worry about losing a file again."

Unless you are running a database it seems like the most common use case for files is write once, update every once in a while. Document creation (code for us or office docs non-coders) would be a bunch of writes up front then follow the same pattern of update every once in a while

I suppose their reads are much less than a typical user, but I didnt think reads contributed much to eventual drive failure

Hypothesizing about how Backblaze stores data: because they likely keep a series of backups, it's more likely they write each delta out separately. I'd imagine it's likely they no not have whole copies of your latest backup on their drives, but they could reconstruct one at will if needed. If that were the case, it would indeed be close to write-once.

Backblaze engineer here. If a file is less than 30 MBytes and it changes even 1 byte we push an entire copy to a new location in our datacenter. We preserve the old copies for 30 days so you can "roll back time" up to 30 days (like if you edit a document but want to revert).

For files larger than 30 MBytes we break them into 10 MByte chunks and only transmit the chunks that have changed. So the worst case is you insert 1 byte at the very start of the large file - this effectively changes EVERY 10 MByte chunk and we transmit it all. The best case is you append 1 byte to the end of the large file, because then we only have to transmit that one chunk.

I'm very curious; has a scheme that attempts to optimize against that worst case been considered? (Assuming you guys have, was it not used because the net tradeoff wasn't worth it?)

It seems like if you're already going through the effort to see if chunks have changed, you can readily do some heuristic where you check the end chunk of both sides, and scan inwards until you find a change; something to that respect, further minimize replicated data? I'd assume you might have CPU cycles to spare, but I'm too far into assuming already to feel comfortable so I'll just hope you answer :P (thanks in advance, it's been great to read what you've written thus far.)

> have you tried a better binary diff algorithm

We just never got around to it. In practice, it turns out most well written programs with large data try not to insert one byte at the start of the file. For example, take your large Outlook "pst" file (commonly 1 - 4 GBytes). When you get an email, it seems to append it to the END of the pst file, plus update some internal tables. So a large amount of that won't change.

Also, the worst case is it wastes space in our datacenter (and some bandwidth) for 30 days and then is cleaned up anyway, so you can measure the theoretical amount of money to save and it won't profoundly change our business so we put it off. Not to say ANYTHING is off the table, we're just always swamped with some project or other. :-)

Remember they do a complete read of the pod to check the data every 2 weeks.

Reply on news.ycombinator.com