Hacker News Reader

Comment by liuliu | Hacker News Reader

What's the best way to store around 300TiB data cost-effectively? Assuming would have access from time to time to around 25% of that data (monthly) through 1Gbps connection.

On-prem seems to be the best, but carrying around 30 disks is no small feat.

Edit:

The data will not be mutated after write. Might append new data occasionally after the first dump.

View on HN

The main questions would be durability and availability. Maybe only because I am scarred by experience but "on-prem" seems to me quite obviously the worst possible way to do it.

Why not both? Store it locally for ease of access, maybe with a solid state cache, backup in the cloud for durability. Storage is cheap nowadays. The more interesting question is availability - at the current HDD sizes, "classic" RAID is not sufficient.

> at the current HDD sizes, "classic" RAID is not sufficient.

Eh, RAID 6 is fine. Your chance of losing two extra drives in a two day rebuild isn't that high if you've been testing your drives regularly.

And if you haven't been testing regularly, even small drives would eventually have some scary risks of dying.

That said, 50% more drives with raidz3 is a lot safer than RAID 6.

It’s not as clear cut as this. Raid6 on how many disks? raid6/zraid2 with 6 disks total might be fine, but 20 disks is not.

One of the more common scenarios with consumer grade hardware is a degraded array getting completely lost while being rebuilt.

> Raid6 on how many disks?

The same number of disks as you were using when the disks were smaller.

I'm comparing like-for-like except disk size.

> One of the more common scenarios with consumer grade hardware is a degraded array getting completely lost while being rebuilt.

Which is why I mentioned regular testing. If your disks are on the verge of death it doesn't matter how much parity you have or how big they are, a rebuild has too much chance of failure. You need to make sure they can handle a moderately heavy load.

But if you're putting them under that load regularly, a rebuild is very unlikely to be the last straw for multiple drives.

As far as I know, even with the recent increase, Backblaze B2 is still the most cost efficient option at $6/TB/mo... but I'll be watching replies closely.

iDrive E2 is $4/TB/mo, that's the cheapest I've seen from a known vendor: https://www.idrive.com/object-storage-e2/

Is there some 3rd party client for their backup service for linux? I was just looking at them for offsite but I have mac and linux boxes to back up, not mac and windows.

What's the access pattern? how often you write and read? and how much?

have you tried getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem?

An FTP account where?

You can create unlimited FTP accounts locally in a config file.

You could probably put the disks in an enclosure and then you'd only have 1, maybe 2 items to carry around.

how often is the data being appended or modified? You might consider a blended solution with the most cost effective on prem-disks w/ no durability or backup, then writing any additions to cold storage which is cheap. The only time you need to access that is (hopefully rare) restore scenarios.

> carrying around 30 disks is no small feat.

20-22TB HDDs are available nowadays, so more like 14-15 disks.

The datahoarder subreddit is probably the best place to ask for this info.

It really isn’t. They are very hostile against people who has been using/abusing google/amazon/dropbox for storing insane amounts of data.

https://www.reddit.com/r/DataHoarder/comments/155so84/dropbo...

https://www.reddit.com/r/DataHoarder/comments/15gf2rc/dropbo...

rsync.net

Reply on news.ycombinator.com