Hacker News Reader

I'm not sure if the reasoning holds beyond, "For the way we designed tarsnap it doesn't make sense."

I agree -- that's exactly what I was saying. The title of the blog post was "Why Tarsnap doesn't use Glacier", not "Why you shouldn't use Glacier".

I think Glacier is a great service, just not a particularly good fit to Tarsnap.

"But if the cost of storage is much much smaller than the retrieval cost, then your design methodology would be different."

If you read between the lines a bit on how the second-to-last paragraph would manifest in code, I think that's what he's already getting at. A Glaiciated machine would probably not get global de-duping, but merely a machine-level de-duping, if even that. (The retrieval cost structure is quite bizarre.)

I think the most important part is separating the de-dup metadata from the data. I assume the de-dup is done via either digest or HMAC -- if you keep the digests (and the blocks of digests, et al.) and pointers to the data location in Glacier in S3, couldn't you do the de-dup using S3 but actual data storage using Glacier?

This is exactly what I've been wondering - obviously cperciva is 10x (100x?) smarter in this area than I ever will be, but I kept thinking, "Just keep the digest in online storage, and the actual data in (offline?) Glacier. You don't need to access Glacier during backups to calculate digests, because you already have the digests online."

I read through his posting carefully, to see if he would capture that possibility - but didn't really see it there.

Think that through a bit :-). So what counts as a 'retrieval' ? Well in the tarsnap case it appears to be a block which as far as glacier is concerned it probably thinks it is a file.

[ NB: huge huge guesses here about how tarsnap works ]

Now consider de-dupe in blocks (vs de-dupe in files) with an object store, combined with a backup. Now lets say you have a file that is 5 'blocks' long. You can get the original 5 blocks (5x fetches) then each block where deltas reside, to recreate the file you want at the time you want it.

Compare that to the 'stupid' way of doing it which is to store a full image of the file system for each time period. Where you have one fetch to get back the file from the copy of the file system that you were interested in.

But this is where the assumptions that make that stupid need to be evaluated. It is a poor choice because the expensive thing is the storage, and you trade compute cycles for storage. That is normally a winning idea because you only 'spend' the compute cycles when you reconstruct the file you want, but you continually pay for storage month after month.

Except that the pricing model of Glacier makes bulk storage cheaper and algorithmic reconstruction expensive.

Presumably the folks at Amazon are de-duping, after all they get to charge per the GB and if they can sell that exact same GB to two people, well that is a win!

So back to the question at hand.

Lets say your middleware layer is just like it is today, figures out just the deltas in your file system from the last backup and then pushes those. You've got a file system image plus a delta image and by applying the delta to the full image you can get back to the current image. However, since storage is now less expensive than compute, at the Amazon instance you take the delta apply it to the latest full backup, and create a new full backup which you then store in Glacier.

So can this possibly make sense? That is the question, if you keep full copies of the latest copy of every file, and pointers to the reconstructed previous versions in your S3 meta data. Can you get a lower net cost for the service implementation?

The deduplication is done on the client side; and yes, it is done by comparing HMACs of blocks.

This would indeed be true for a sufficiently large price advantage of Glacier over S3, but note that tarsnap typically saves ~99.99% by de-duplicating my data (daily backups, usually only a few e-mail and code files have changed.) I imagine that the typical usage pattern may be a bit less extreme, but the price differential would have to be huge to be worth it.

I don't know if only sending the changes is considered "De-Duping" (though it is related) - That's more equivalent to doing differential (or block level differential) backups.

De-Duping is if you add the same file (or portions of a file) in 10 different directories - the data is only stored once in backups.

Dropbox takes it a step further - if 1,000 people store the xcode DMG on their dropbox store, only one copy is stored in Dropbox.

I don't know if only sending the changes is considered "De-Duping" (though it is related) - That's more equivalent to doing differential (or block level differential) backups.

The way Tarsnap does things, it's the same thing; Tarsnap splits data into blocks and deduplicates those blocks without caring whether each block is a whole file or part of a file or several small files stuck together.

When calculating dedupe ratios you should be careful what baseline you use. Performing full backups every day is generally not realistic. I would compare dedupe-based systems against classical incremental backups.

Based on the article it seems like the motivation for the deduplication is as much bandwidth conservation as reduction of storage costs. What you're proposing would make sense if storage costs are your only concern, but it shuts out bandwidth limited users.

An excellent point. So capturing the bandwidth savings and the storage cost savings suggests an interesting mid-tier between longer term storage and shorter term.

There's no bandwidth cost associated with transferring data TO Tarsnap (that is, data transfer into EC2/S3 is free of charge), and if you have to get a file out the cost is the same whether you store files or "blocks" (I'm assuming the block disassembly is done on EC2).

So our hypothetical Tarsnap alternative could store file metadata in S3, recently modified files in S3 also (quicker retrieval) and push older files into Glacier (can be retrieved when needed, cheaper storage if (as is likely) never accessed).

It will take marginally more time and bandwidth to transfer a file every time there is a change than to only transfer changed blocks (i.e. parts of files that have changed) but for 90% of users I would bet the (significant?) decrease in long-term storage cost would make that worthwhile.

If you read the article, you don't have to assume (incorrectly) that the client sends the whole file for each backup. Indeed, a significant motivation is to avoid sending very large files across bandwidth that is constrained (or costly) on the client side.

> I currently have about 1500 such archives stored. Instead of uploading the entire 38 GB — which would require a 100 Mbps uplink, far beyond what Canadian residential ISPs provide — Tarsnap splits this 38 GB into somewhere around 700,000 blocks, and for each of these blocks, Tarsnap checks if the data was uploaded as part of an earlier archive.

That's not what I assumed at all, please re-read my comment.

I'm saying that if it meant using Glacier was possible, doing delta-backups on files, rather than blocks, might not be a bad tradeoff.

"There's no bandwidth cost associated with transferring data TO Tarsnap"

Except for those of us who's outbound bandwidth isn't free and/or unlimited. It makes no difference what Amazon charges Tarsnap for it - I can't upload all of my 128G SSD every hour - I just don't have the bandwidth to do it, and if I _did_ have the bandwidth, it'd probably send me broke pretty quickly (or have my ISP throttle me or cut me off).

It's not a Tarsnap cost, but it can be a Tarsnap user cost, and I suspect cperciva considers that just as much a "real cost" as actual monetary expenses to Tarsnap.

The point of a dedupe-based service like Tarsnap is that while logically you upload the whole disk, physically only the changed blocks are sent to Tarsnap.

It is extremely convenient.

Well, it's possible to make the deduplication work with Glacier, though. Colin's right that you want to phase versions of the whole tar into slower and cheaper storage, but the technical problem of whether blocks are used in newer versions doesn't actually seem to be too much of a problem. You can compute it in a batch for extant tars, and track it for new ones. (And you can keep blocks-of-references always in S3 rather than Glacier, say.) The problem he identifies as most troublesome is what if you see a case where you would want to deduplicate, but the similar block is in Glacier, and so would be a bottleneck for the whole deduplication process. In this case, you could always treat it as a non-match, right? And optionally have some way to track it and then when it gets moved to Glacier either determine whether it was actually a match all along, or just have some duplicate data.

Of course, I don't know how tarsnap names its blocks or stores them, so I don't know how feasible it is to have two blocks with the same name because they had the same hash but there was a byte-by-byte mismatch, or if that's even a problem.

I mean, blocks that have been moved to Glacier because there are no references to them from indices on S3 can be assumed to be less likely to show up in new archives. It's a trade-off, but my experience with deduplication is that it's often not much of a trade-off to get rid of old things even though the magical thinker in me is tempted to think "but what if that chunk happens to show up again somewhere else!?"

Glacier is simply meant to solve a different problem than S3.

I will be using Glacier for storage of litigation hold data -- potentially many terabytes of stuff that is being held because a plaintiff asked to hold "everything" and a judge agreed.

So I'll pay to store this stuff for a few years, and in the off-chance that I need to retrieve it, the other party will pay at least half. (A powerful incentive for them to not ask for the data to be retrieved in the first place!)

Basically, you want the data that your sending there to be contiguous -- no fancy de-duping or incrementalism. If you are going to use it for backup, just tar up everything and send it up in blocks that make sense from a recovery cost POV.