I buy drives in pairs, and RAID them. If you can't afford to do that with drives the size you need, buy smaller drives and keep less data. It's like being a half-decade behind on the Moore's Law curve. You get used to it and survive.
Using a filesystem that supports data scrubbing doesn't require an "enterprise class" budget anymore. After fielding this question from several of my beta users, I wrote it up the why's and how's on my blog: https://photostructure.com/faq/how-do-i-safely-store-files/
I haven't run into much bit rot yet, at least where it mattered, but I'm guessing I'd see more of it if I went up to bigger drives. For now, my data lives on 4tb.
PhotoStructure sounds neat. Business model may be tough. I wouldn't run a proprietary solution for data management, especially from a small startup. I just wouldn't trust it. I want to know what happens to my data, and to be able to get it out. I understand the claim you'll open source in case of business failure, but there are many other paths (e.g. bought by Oracle, milked for profits, moved to incompetent management, etc).
I would totally would pay for an open solution, if there were a good business model for that, but I'm not sure what that is. I'd pay for integration (a piece of hardware arrives on my doorstep with no futzing, and with automatic security updates, etc. and your competitive edge is trust).
I'd also consider paying for a split model, where the data management used open formats and auditable, documented open source, while there were UI/UX or search functionality on top of that. That may be what you're doing; I'm not sure. I can see some open source stuff, but I'm not sure what's open and what's closed. Or better yet, where there's a cloud model. My data lives at home, but you have low-res versions of my photos which automatically sync to my phone, which can be shared, etc. securely.
Mostly, what I want is a place to reliably be able to dump my files (not just photos), have versioned backups of everything, have an automated way to move to offline storage once a quarter as you recommended, and have some way to manage !@#%^ duplicates (I have several computers, and that makes for all sorts of messes). Ideally, I'd also have some way to securely share with people
https://www.redhat.com/ja/blog/what-bit-rot-and-how-can-i-de...
All three are available to recent linux distributions, but know they all have very different ergonomics. Spend a minute or two looking at a getting-started guide for each, and pick (or if you have a tech friend that might help you, pick whatever they recommend). All of them work.
> I want to know what happens to my data, and to be able to get it out.
You may have missed this, but with PhotoStructure, none of your data leaves your computer: it's all sitting in plain files and in a schema-documented SQLite database in your library.
> ... I'd pay for integration (a piece of hardware arrives on my doorstep with no futzing, and with automatic security updates, etc. and your competitive edge is trust).
That's an interesting idea, but I think logistics would prove challenging. There are a couple startups that have tried this in recent years, and failed.
> I can see some open source stuff, but I'm not sure what's open and what's closed.
The metadata parsing code (exiftool-vendored) and process management (batch-cluster) is open-source. The metadata inference and cleaning, image duplication detection and comparison, and UI are closed-source.
> Or better yet, where there's a cloud model. My data lives at home, but you have low-res versions of my photos which automatically sync to my phone, which can be shared, etc. securely.
I think there are already cloud solutions (like google photos) that provide this. I think PhotoStructure's value proposition is in privacy you only get via self-hosting.
> Mostly, what I want is a place to reliably be able to dump my files (not just photos), have versioned backups of everything,
borg, tarsnap, and BackBlaze b2 are good tools for this.
> have some way to manage !@#%^ duplicates (I have several computers, and that makes for all sorts of messes).
Simple SHA collisions are easy, and there are many command-line tools that accomplish this. PhotoStructure does detect SHA-duplicate images, but also downsampled, RAW+JPG, and other variations: https://photostructure.com/faq/what-do-you-mean-by-dedupe/
> Ideally, I'd also have some way to securely share with people
This is coming in a future version of PhotoStructure.
Tech friend (yes you!): Which one do you recommend? I mostly care about avoiding headaches (so reliability, stability, and ease-of-use).
> You may have missed this, but with PhotoStructure, none of your data leaves your computer: it's all sitting in plain files and in a schema-documented SQLite database in your library.
I got that, but my goal is to avoid headaches. If your business goes under, I don't want to lose a couple weekends to data recovery.
I think open source APIs to get at my data would be the level which would give me confidence. If I have access to the same APIs as your UI/UX uses to get at the data, I'd be okay.
> That's an interesting idea, but I think logistics would prove challenging. There are a couple startups that have tried this in recent years, and failed.
I wouldn't look at other failures as ways to discredit an idea. Most failures are due to poor execution. That said, hardware business models are very different from software, and if it's outside of your comfort zone, you'll probably fail to execute too. But my point was:
1) There's a definite business proposition in providing a complete solution to the problem you articulated with very close to the technology you're building. You seem to be on the right track.
2) I'm not sure you've stumbled on the right business model yet.
I want a solution which minimizes headaches, both current and future potential. I'm glad to pay for that.
> I think there are already cloud solutions (like google photos) that provide this. I think PhotoStructure's value proposition is in privacy you only get via self-hosting.
Well, in part. I think the other pieces are:
1) Reliability. Google can and does shut down accounts regularly. It loses data too. I'm a statistic.
2) Google downscales images and doesn't handle ones from my big cameras very well (it's great for cell phone picks).
3) Storage. I can buy many TB of disk space. Many TB in the cloud is still expensive.
4) Trust. If you build a tool which (a) lines up incentives right (I pay you) (b) has the right legal code (my data is mine, and unless you receive a subpoena, stays mine) (c) has the right technology so I only trust you when I absolutely have to (I can share files with end-to-end encryption, or openly, depending on what I'm doing)
5) Integration. I'd like RAW files and similar to live locally. All I want in the cloud are pics for showing people on my phone, and nice search tools. I want metadata synced, though.
What you're building is a lot more useful than anything else out there, by design, I think.
> borg, tarsnap, and BackBlaze b2 are good tools for this.
Thank you. A few points, though:
1) These do backups. I want synced file management WITH backups, not just backups. If I edit a Word file on my Mac, copy it to my Windows box, edit it again, and look 6 months later, I have a mess. If I then get rid of both machines, and buy a Linux box and copy both files over unto a RAID, I have a mess with duplicates.
2) Tarsnap is $0.25/GB/mo, so $250/TB/month. a 4TB backup would be $12k per year.
Just about the only thing I feel I have well-organized is my code, which lives in git. And a lot of my documents, I author in Markdown or LaTeX, and those live in git too.
It is god damn annoying.
Sure it can. You use mdadm to schedule a monthly full scan of the RAID array, comparing both disks. It's the default in Debian.
You should also use smartmontools to schedule weekly long self tests of the disk, and nightly short self tests. That way you find out about problems when they are still small.
We had backups and could replay all db transactions up to the minute. Still, the service stopped for more than 24 hours. It took a week to clear the backlog.
Make it an OEM problem.
Those were the heroic days.
First, while RAID isn't a "safeguard", the idea that there is a high probability of raid failure during a rebuild has been repeatedly proven to be false as long as proper data hygiene is maintained. For the most part the chance of a second disk falling out during the rebuild is exactly the same as any other day regardless of capacity.
Put another way, if your scrubbing (or just accessing the data) frequently enough to avoid having silent bad sectors the idea that there is a high probability of a drive loss during the day or two it takes to rebuild an array would also result in drive losses on a nearly daily basis.
So unless your constructing a single RAID5 array out of ten thousand plus disks, if the MTBF numbers are anywhere close to realistic, drives in a single array aren't going to be failing at a rate high enough that the chance of a double fault not caused by something symptomatic across the entire line of drives (say a firmware that wipes the drive after X power on hours, in which case no level of redundancy will save you) is a real worry.
So, the rumor is based on the idea that your increasing the RAID rebuild time from a couple hours to days/weeks on end due to the larger capacity. But this is also false, because as density increases so has the sequential throughput of the drives. AKA comparing a ~ten year old 4TB drive doing ~100MB/sec vs a modern 8TB drive doing 200MB/sec. There are some non-linearities due to the number of platters and the rate of TPI vs BPI increases but generally platter count is somewhat constant and TPI and BPI are both increasing. Now if your hitting internal RAID limits because your running modern drives on old controllers which cannot rebuild at the drives peak throughput that is a different problem.
Bottom line, your doing it wrong if RAID5/6 isn't increasing both actual as well as theoretical availability. The MTBF numbers are published to give a baseline on what the expected failure rates are.