What you had was a single copy (probably a simple rsync, am I right?)... that is worth nothing in case of corruption or voluntary mischief.
Learn from your mistakes. Either use a solution that has a retention policy, or don't do backups at all.
The way apologists step up to the defense of stuff like this is appalling. The 'hard' stuff like this is supposed to be dealt with before the public release. Just because one has a backup doesn't mean that other programmers should force them to rely on it (whether they intended to or not... the point is that this kind of thing should never happen)
I am beginning to feel like we should start figuratively burning people at the stake for advertently or inadvertently destroying other peoples' data -- it is a sin about as cardinal as they get; and it seems many programmers are clearly not very scared of such an outcome. Wiping/bricking/rendering inaccessible your customers' data should be causing people to shit bricks, not go on the defensive.
Normals HATE technology bullshit like this. They just want it to work. Failures like these make the entire field look bad.
But in the end, you can't trust them to not fuck up. You have to trust that they will make a mistake, so be prepared for it. Doing backups right is just really hard.
As nice as that idea is, I don't think we have the tech yet to fully trust someone else with your data. You have to test it out that your backups work. Anyone who tells you otherwise is selling snake-oil at best.
Which, I think, was alexggordon's point. That he no longer wanted to manage the data himself because it was a hard problem, better managed by teams of professionals who have the time to do things like verify backups because they can amortize the cost of that effort over tens of thousands of customers.
Judging by the frequency of security breaches they don't hate it nearly as much as we would want.
As a Systems Librarian I did the daily (31 days worth), weekly (52 weeks worth), monthly (48 months worth) back ups with 131 tapes was a pain but it always paid off. I just had three machines and needed to go to my tapes once a day.
Insider information all the Library Management Systems are totally broken and garbage (Open Source Evergreen is doing things better). I was required to reboot my Windows Server (Purchased before I was hired) everyday. I was told this was totally acceptable. The Oracle DB was solid but the rest of the stack was total and complete garbage that cost over $125,000 and service contracts of $30,000.
Edit: Off Site aka IT Office Fireproof Case was my monthly and previous weeks. Once a week I walked them over. In the library in a fire case were the daily, and the last week. Tape was it was 2007.