Hacker News Reader

Comment by perlgeek | Hacker News Reader

So, let me get this straight:

* It's been deleted for a week already, they estimate they might need two more weeks. Three in total.

* They claim to have "extensive backups", and hundreds of engineers working on it.

What? How? This simply doesn't go together. Why would restoring from backup take three weeks?

Either their backups aren't complete, or they need new software written for the restore, or something else doesn't add up.

I haven't administered their software yet, but what I've learned from the sidelines, at least Jira doesn't seem to be rocket science. A database, an application server (maybe a few instances for larger sites), a bit of config, some caches. This really shouldn't take three weeks to restore.

View on HN

If you would permanently delete data for selected customers from a large multitenant system, it could actually take some time to restore it - even with proper backups.

You can’t just do a full recovery as that would mess those customers who were not affected (it likely takes time to notice the mistake - others have continued to use the system). You might need to write some tools to migrate the data from backups. Also you really need to test everything very carefully - otherwise you might be in even deeper trouble (looking at corrupted instead of lost data).

In large organization this kind of ”manual” recovery might require people from multiple teams as no single person knows all the areas. This adds overhead. Throwing too many people in does not help either. When you start thinking about it, few weeks is not that long.

And JIRA is definitely not simple. It’s complicated beast and likely the SaaS features combined with all the legacy makes it even more complicated.

What a nightmare situation. Makes you wonder if some kind of 1 database per customer setup would be preferable here since you could restore only affected customers.

how would that require hundreds of engineers though? Would one not build a script and then just run it for each customer or each dept build a script? I honestly have no idea, never been in that sort of recovery situation before.

I was in a similar situation many years ago (different ticket software though). What we did was spinning up a spare server with the backup data and script an extraction/injection tool to populate the production multitenant sass.

Restore from off-site tape backup. The kind of service where you ship them ~dozen new tapes in a lockbox each week and they ship you the oldest dozen back. It's supposed to be the "if all of a data centers happen to burn to ashes simultaneously" option. If you say "give us all of our tapes, asap" and then have some pour souls swapping them out as fast as the data can be read... it would probably take a few weeks.

Tapes are not that slow, 360MB/s per drive, and on a large scale swapping them is completely automatic.

In support of your point, 360MB/s is an extremely conservative estimate. I'd expect that from LTO-6, which is around ten years old, and I would certainly hope their backups are on more modern gear than that.

I work for a pretty huge well-known fortune 500 company with a global presence and personally handle the tape rotations for several of our data centers, and they're all still on LTO-6 tapes

I'm not sure that's relevant. I would expect Home Depot, Berkshire Hathaway, Alphabet, and Atlassian to have wildly different priorities. Two of them are Fortune 500 companies that I think would do fine with LTO-6. Atlassian is supposedly a cloud-first organization which arguably should be pretty aggressively tiered anyway, much less tiered onto outdated tech.

And that's just one drive. I ran a piddling departmental backup server and we had a dual-drive library.

Four drive libraries are pretty common, too.

My theory in my other comment is that they've deleted some data and are waiting on third-party data recovery specialists. That would explain the timescale.

Jira cloud became much more complex when they went all in on AWS. The reason for the cloud/server fork 4 or 5 years ago was so that cloud engineers could couple to a zillion AWS services without having to build back any backwards compatibility. So data stores are very much more disparate than just a PgSQL DB and redis (which is how it used to be).

Restoring everything from backups is very hard.

Something that can make restore-from-backups harder, and that I've seen happen, is when the backup/restore systems themselves get destroyed by the same black swan event. Then you have to first recover those by doing fresh installs, and you have to have all the people on hand who know what the configurations would have been to be able to then use the backup library. Then you have to begin restoring a few target systems to check that everything is OK with the restore process, then you have to restore everything though you'll be limited by the restore system's bandwidth.

How could this happen? Well, a disgruntled employee could make it happen. It happened at Paine Webber in 2002 [0]. In that case the attacker left a time bomb in the boot process on all systems they could reach, and that included the backup/restore servers. Worse, the time bomb was in the backups themselves, so restored systems ate themselves as soon as they were booted, which slowed down the recovery process.

  [0] https://www.independent.co.uk/news/business/news/disgruntled-worker-tried-to-cripple-ubs-in-protest-over-32-000-bonus-481515.html
  
      https://www.justice.gov/archive/criminal/cybercrime/press-releases/2002/duronioIndict.htm

If you are not regularly testing restores then you don't have backups.

I'd say this is the answer.

I remember a situation where we had a near miss with data loss (replica failed and master had a bad disk). We didn't want to put the production database under extra load by taking a live backup while it was handling all production traffic, so we restored a backup. But it was "bad". Tried the one before it, and the one before that. Apparently they were busted for over a month due to a config change. We restored a month-old backup and started applying binlogs (which thankfully we had been backing up). But that meant replaying a month of transactions into the restored database. I can't remember the details but I think we ended up replacing the bad disk, resilvering the array and live-cloning the primary before the binlogs got fully applied to the one we restored from the old backup.

Went through that once in the mid- late 90’s. Each restore and test took hours, so 3-4 attempts took 2 days - with me sleeping, crying and praying in a conference room.

I have no specific information on this, but in general and in theory:

Isn't this how an incompetent, insincere and desperate company being subjected to a ransom attack would communicate publicly?

My guess is they failed halfway through a major schema or api migration. If some of the services have already progressed too far, then rolling back another service to previous backup snapshot will make the two incompatible. Especially if one of the services is global and the other is per customer.

The only way out is to figure out the bugs and continue migrating forward, fixing issues as they appear one by one.

What if they have backups of the data, but not the specifics of users' sites?

Reply on news.ycombinator.com