Microsoft's Sidekick/Pink problems blamed on dogfooding and sabotage
appleinsider.com
appleinsider.com
Also, this article seems to be about 98% speculation from a site that has no previous history of "inside sources" at Microsoft. Boo on that.
(Not to suggest that Google doing it makes it right or wrong.)
I love how the author takes the fact that Microsoft/T-Mobile haven't issued an official statement on the cause of the outage, and decides that it therefore MUST have been caused by an elaborate and sophisticated act of sabotage by some ne'er-do-well engineer.
QED.
Inevitably acquired companies will wind up porting their software to whatever insane crap their business unit is using as a development stack. We're about to start going through that pain right now.
Can anyone quantify how much address book data there was? As an outside guess, 1,000,000 customers times 1000 addresses times 300 bytes each is only 300 GB. They could have kept an emergency backup on rsync.net for $300 / month. I keep secondary, well-encrypted backups of things I particularly care about (not nearly 300 GB worth) there, as well as on a USB disk under my bed, in addition to the regular complete backups.
What about off-site backup? How many failures can cascade in a way to wipe out off-site backups?
Or not even offsited, it's not like the building burned down. Just tapes not left in the library.
The theory that they might have the data but just don't know how to restore things because all the Danger people are gone, now that actually seems somewhat more credible to me.
Because SANs allow multiple machines to write to the same disk, they have byzantine failure modes where data can be overwritten on disk, but cached for long periods of time by the machines that need it so there's no visible problem until after a power failure.
When data is migrated to new database hardware, sometimes the backups haven't actually been tested. Sometimes during the cutover, someone who didn't fully understand the architecture backs up the new (empty) machine on top of the old machine's backups, so there is no backup for a while. All these mistakes can happen pretty easily when sysadmins are woken up in the night to fix a service outage while people are yelling at them. I have made some of these mistakes myself, though so far I've been lucky.
The computer industry has never learned to admit that there is always some level of risk. You won't hear an oil company exec say "Inconceivable!" when an iceberg crashes into a North Sea oil rig, or a refinery catches on fire. Systems fail, and all you can do is try your best to avoid it and move forward after it does happen.
Some of the things like 'locking the keys in the car' can be mitigated by making sure that you test you backup system.
Obviously you can't prevent everything, but to say that some of these issues are 'unavoidable risk' is like saying that an oil company exec would say, "shit happens," to reports that one of their oil tanker captains crashed a ship while drunk.
The real problem is that backup is usually an after-though.
Once upon a time, I wrote a series of opensource iSync plugins for the sidekick/hiptop platform, back when Danger first opened up their XML-RPC Service.
Then it came out that in order to run in "production" (e.g. on a phone without a developer provisioning loaded) your app had to have t-mobiles permission. We kept repeatedly getting told "no" on the iSync plugin, that they had no interest in supporting Mac. Then of course a commercial plugin appeared and I got fed up and stopped trying, getting the clear impression that commercial was what T-Mobile wanted.
Mostly however, it was the shock in realizing that you had no access to your own data as a standard user. You were completely and utterly locked in to that device, with no alternative.
Well done AppleInsider, you managed to nail the problem simply by covering every possible base.
Microsoft has a hardon for dev-managed directly-addressed OS instances running on x86 machines, so you could at the very least migrate to their extant hosting infrastructure. Even if you have to rewrite your app in C# + SQL Server, it's still going to be a direct gloss for most traditional webapps.
Google does no such thing -- everything is massively distributed at every level, where blocks of infrastructure are managed as ideal services independently from any application, and addressed at the datacenter level. You are not going to get to manage your own machines, access a traditional filesystem, use relational databases, a direct socket to the client, direct access to internet hosts, basically anything you take for granted.