Hacker News Reader

Comment by pgrote | Hacker News Reader

Textbook on how not to handle the issue. Deleted Twitter, sanitized Facebook and more than 2 days to admit the issue.

There is discussion on twitter that the company said the backups were on the same network as the data. Hopefully there is an offsite backup available.

https://twitter.com/ConleyU/status/1151862278909825024

https://twitter.com/MRasconCPA/status/1151894366291734533

https://twitter.com/hockeygirlPDX/status/1151945932935585792

Ouch. This is the sort of stuff that can kill a company.

Does Quickbooks with the cloud option offer local backups?

View on HN

It is interesting how often I see companies that refuse to make back-ups of their cloud hosted data on the assumption that this is now someone else's problem. I also have a - recent - case of a very large manufacturer of storage solutions that managed to fuck up a restore of a raid array to the point that a whole pile of companies lost their data.

Backups are so simple, and yet the only times people seem to realize their true value is when they don't have any.

I am not arguing the importance of backups, but tbh, simple is not a fair statement.

At the very least, if you have backups, you might have the ability to rebuild/redeploy, but for any non-trivial system, without a comprehensive DR plan that is drilled regularly, even having backups won't save the business if the tooling to get those backups into a production ready state.

It's also a major flaw of the cloud and SaaS providers. They don't encourage customers to maintain local (or alternate-cloud) backups, because they refuse to admit the appearance of weakness.

If a Cloud/SaaS provider is going to claim to be your hassle-free one-stop shop, shouldn't they provide, as a matter of course, a feature to dump your data daily/weekly/monthly to a designated off-cloud site of your choosing? But they don't, because they want to pretend that they are impenetrable and 100% available.

Would be nice if someone could build a site like https://twofactorauth.org/ for bringing more attention to this alarming and growing issue.

The goal to hopefully influence companies into providing such a fundamental ability.

A data dump isnt much use if you don't have the software to use said dump. Most SaaS providers don't run open source software...

> Backups are so simple

It gets more interesting when you're trying to backup a distributed database. When your durability strategy is to spread your risk across a lot of nodes in a cluster, and you're federating your cluster across multiple sites, traditional backups may not be a practical option. However, as I write this, I'm realizing that there's nothing simple about distributed anything, and saying "it's hard to..." may not be all that interesting.

Pick a data sequence number, and back up each region sperately at the same sequence number.

Keep your backups on-site but off-line - you already have many locations. You should practice a location (and it's backups) going offline at the same time as a code bug screws with data in the active system.

Cassandra handles this pretty nicely. Files are immutable, so files are hard-linked to a snapshot directory. Copy out the snapshot directory, delete it, and things are eventually consistent.

We restore our MySQL and Cassandra data every day in our production DC from backups in the DR DC. It's the only way to ensure things are working. It's about 100TB of data, but things zip along nicely now that 100gig networks are (more) common.

Neat!

> Backups are so simple

Disagree. A high-confidence backup plan that includes continuous restoration is of middling difficulty to design. Many get backups wrong and only find out when their archives turn out to be incomplete.

Backups are NOT simple, especially when you have things like GDPR in play. I'd imagine companies can't afford to commission backup that is compliant with the regulations. If you spend on a backup system, your company goes down in debt, you don't - there is a chance you go down anyway.

If you can not provide a backup system for a fraction of the costs of hosting the production data - including GDPR compliance if that is what is required - then your company does not deserve to be in business to begin with.

"But we have a contract saying that it can't happen" or "but we have a contract saying it's their problem"...

(Which works reasonably well as long as the damages are quantifiable and the other company is actually able to pay them....)

QuickBooks Online does not have a native backup or restore, at all, and it's pretty ridiculous. QBO told me they also can't do on-request restorations with their backups.

There's a third-party or two that can use the API to do a backup/restore, but also, still, not all of the QBO data has an API. Eg, IIRC, recurring transaction tasks.

Just a matter of time for ransomware to replace data via APIs.

> Just a matter of time for ransomware to replace data via APIs.

I asked about this in a cybersecurity meeting at an investment bank a few years ago - how much security is there around Bloomberg's data feed? All of the algorithms that make million & billion dollar decisions rely on that data so I asked if someone stepped in the middle and fed the algorithms bad data or even slightly delayed data would we even know it had happened? They looked at me like I was a crazy person. I'm actually kind of surprised it hasn't happened yet. Or maybe no one has noticed yet.

That's a fascinating scenario. Even more malicious embedding shellcode in a series of tickers and prices kicked off with a few invalid tickers/prices.

I wonder how many other feeds/sources like that in which so many businesses rely on in at least some level of autonomous fashion.

Reminds of the story of Google wiping out all of their CDN servers by Diskerase because of a single detected value, IIRC.

As the resident tech guy in a small shop with Quickbooks, yes, you can do local backups. We're not using iNSYNQ for cloud hosting, though.

Reply on news.ycombinator.com