Comment by gjm11 | Hacker News Reader

Comment by gjm11 | Hacker News Reader

While we're discussing making Tarsnap more user-friendly:

The single reason (other than, er, fecklessness on my part) why I haven't tried Tarsnap is that it seems difficult to predict how much using Tarsnap would actually cost me. I would need to know (1) how much space my data will take up after compression and deduplication, and (2) how much bandwidth my incremental backups will need -- again, after compression and deduplication.

Wouldn't It Be Nice If there were a "predict my Tarsnap costs" tool? You download it and point it at your data. It compresses it and identifies duplicated blocks, and says "You would initially be storing about 25 GB of data on the Tarsnap servers, which will cost you about $6.70 per month." It records a bunch of hashes. Then, a little later, you run it again. It identifies changed blocks and compresses the differences (or something), and says "If you do an incremental update like this, you will transfer about 2GB of data, which will cost you about $0.54." With, each time, a disclaimer: "This is only a crude estimate and if you think Tarsnap, Inc., will be in any way bound by it then you're out of your mind."

I'm not sure whether this is a thing that any random third-party person could write, or whether getting the numbers right would require information about exactly what happens on the Tarsnap servers that only Colin has. Perhaps the answer is that doing it right requires secret information, but doing it well enough (e.g., getting within 20%, 95% of the time) is easy with some naive algorithm like "divide everything into 4kB blocks, hash them all, identify duplicates, compress individual unique blocks with gzip; do the same for incremental updates but ignore blocks whose hash hasn't changed".

I'm curious: is this a factor stopping other people signing up with Tarsnap? Is it perhaps only a factor for cheapskate individuals like me, and not for the larger organizations that probably represent most of Tarsnap's profits?

how much space my data will take up after compression and deduplication

Valid point. You can use tarsnap's --dry-run option (along with --print-stats) to find out how much space your data will take up; unfortunately you need to have a key file before you can run that, and you can't create a key file until you've created a tarsnap account and added money to it.

I will be adding a mechanism to allow people to run "keyless" dry runs so that you can install tarsnap and find out how much space your data would take before you spend any money.

Excellent!

Would I be right in thinking that, if the storage cost is small enough not to bother me, then the bandwidth costs are extremely unlikely to be a problem?

The bandwidth used to upload data is going to be around 1% more than the size of the data (protocol overhead). So to a close approximation the cost of the bandwidth for uploading is going to be the same as the cost of a month of storage.

That's for the initial upload, right? (And then incremental updates should typically be much smaller.)

Yes. (Although even with incrementals, until you start deleting data your total bandwidth usage to date will be roughly the same as your current monthly storage.)

The thing that's keeping you from using Tarsnap is its costs? The deltas you're talking about look like they are under $10.

It sounds like you are a programmer. At an hourly rate, it probably cost you more just to write that comment.

No, predictability of costs.

The deltas were completely made up. If I knew they were right, I'd sign right up today. (I think.) What I don't want to do is guess at the deltas, sign up for Tarsnap, start up my first backup or my first incremental thereafter, and find I'm on the hook for hugely more than I originally budgeted.

I think the way it actually works, what would happen is that after my first backup completes my account would go overdrawn and I would lose access to Tarsnap. And then I could walk away and all I'd actually lose would be whatever I'd prepaid. But -- and this is surely irrational, but I bet it isn't only my brain that works this way -- I would then feel really bad because (1) I would be failing to pay for something I had bought and (2) I would have spent money and got nothing in return. Even though (1) Colin surely budgets for a certain amount of attrition and (2) the sum of money involved needn't be bigger than $5.

I don't want to do something that seems like it has a substantial chance of leaving me feeling simultaneously guilty and wasteful.

I did the math and I see your point. I'm also a CrashPlan user.

Right now I'm paying CrashPlan about $60/year. My best guess at Tarsnap costs is about 10x that, $600/year.

Normally I'm a very privacy-conscious person, but that difference does give me pause. I could probably slim down what I do with Crashplan to only personal files rather than the whole disk, but why would I want to do that? The whole point is to "fire and forget", and restore painlessly.

Perhaps Tarsnap makes sense if you're Stripe, with lots of sensitive data to store, easily partitioned from your other data. But if my calculations are correct, probably not if you're an average individual.

Does CrashPlan dedup and compress the data? Also I have to say, tarsnap is so much more lightweight than CrashPlan. If they came out with a nice command line client I would have to reconsider them.

When I last tried to do a calculation, I came out at well over $100 per month which, yes, I can afford, but is way more expensive than what I'm paying Crashplan. There are many reasons I'd rather use Tarsnap than Crashplan, but it's hard to justify that price difference.

This might be one of those situations where the simplest and cheapest way to figure out if something will work for you is to just try it for a bit. You can always stop, but right now you're at risk of data loss. Perfect is the enemy of good, worse is better, etc.

Consider: Your current approach is basically to model the problem, evaluate the model, and then decide whether to use the service. That approach seems to protect you from risk, but it really doesn't. As you've noted, there isn't an existing model, which makes the modeling-first approach too expensive. As a result, you don't have backups, which is riskier that one or two months of abnormally expensive tarsnap service.

Your deciding factor is probably the long-term cost. You can ballpark that by paying only a month or two and extrapolating. That should relatively cheap in absolute terms. Most numbers are not outliers, so you're unlikely to be very surprised. You would probably know already if you're atypical in some way. Even if it turns out you'd have an abnormally high cost, figuring that out by evaluating the service will be cheaper that building a model.

Perhaps you're worried that the costs are fine right now but will grow too high over time. If so, that'll happen relatively slowly, over the course of a month or two at the fastest. At worst, you'll have to delete some backups or switch to a difference service. That might be annoying, but it won't be overly expensive and it's unlikely to even happen. If it does, at least you will have had affordable backups in the meantime. Right now, you're at risk.

Are you sure something like Backblaze isn't a better fit for you, then?

"Unlimited" backups at $5/month/computer. Install their proprietary agent on your Windows/Mac OS X machine, log in, and forget about it. No Linux support, though.

Absolutely sure. The machine I am most interested in backing up is a FreeBSD box.

> (1) how much space my data will take up after compression and deduplication

In my experience, deduplication typically doesn't do much for the first backup. Further backups take approximately as much space as usual incremental backups. So the answer to this is "as much space as your existing backup".

> (2) how much bandwidth my incremental backups will need

I recently looked into this and for me the cost of bandwidth used for daily backups is about 10-15% of the cost of data storage.

Just by way of information for Colin (though I'm not sure how much information it really gives): this comment is currently on +25, which to my mind suggests that there are a reasonable number of others who would also like this feature.

I think the ability to do a dry-run upload without prepaying (which Colin has already said is on the way) will be pretty much a complete solution.

Yeah, I'd much prefer a $10/month for 10 GB of storage plan without any variable pricing.

But honestly, it is what truly stops me from using Tarsnap. The part that stops me is the fact I have a dev server at home I have on 24/7 and I can just wait 9-10 hours for it to pull gpg'd backups.

It will cost you approximately nothing.

Reply on news.ycombinator.com