Hacker News Reader

Comment by inopinatus | Hacker News Reader

There is a tale - perhaps apocryphal - handed down between generations of AWS staff, of a customer that was all-in on spot instances, until one day the price and availability of their preferred instances took an unfortunate turn, which is to say, all their stuff went away, including most dramatically the customer data that was on the instance storages, and including the replicas that had been mistakenly presumed a backstop against instance loss, and sadly - but not surprisingly - this was pretty much terminal for their startup.

View on HN

The third largest bitcoin exchange made a change to their RAM settings in EC2. This shut down the machines, wiping out the hard drive and RAM. Their wallet was stored there. They lost everything.

https://siliconangle.com/2011/08/01/third-largest-bitcoin-ex...

Third largest bitcoin exchange...in 2011.

Well the bitcoin supply is limited, so it's quite a big loss.

For them! Due to the way the economy works, destroying money is actually a donation to everyone else who has money (deflation).

Still hard to imagine this wasnt ever just laughed out the room immediately

It's infinitely divisible, so not in the same sense as the loss of any other limited item. There could be only one Bitcoin in existence and it wouldn't change the utility of the system.

How do you send half a sataoshi?

Get all the verifier nodes to agree to increase the divisibility of BTC into sub-satoshi (e.g. via a BIP request approval). This will not be necessary for many generations.

...with 17,000 BTC.

(Valued at 220k USD then, or 700M USD today)

So funny reading comments from that era to the articles about this incident. Some say they were lucky because they are using Mt. Gox! Another observation - bitcoin at that time was by many perceived as and had reputation of in-game purchases.

> This event will go down in a long litany of Bitcoin events that have caused massive losses in the BTC economy.

Reading this from 2011, and reading the latest on Web3IsGoingGreat.com today, is vaguely staggering. They never have learned.

Ooof... One copy and inside an instance? This person should not have been handling money.

How..? Was there no local code on any of the dev machines? No git? I'm asking because for example if Github is vaporized today, my product would lose roughly a day or two's worth of work, since we have like 30 computers having a repository copy.

Of course redeploying every single thing would not be seamless because of course, there might be some configuration stored in services, or something similar, but I'd say that ~90% of our automation is stored in Git.

They lost customer data, not source code. You shouldn't have a local copy of all user data on your machine.

But my customers enjoy the personal touch of me manually editing their SSN in my local MS Access database.

I mean, after all, if you don't have your own copy of the MS Access database then when your team scales beyond about 5 people that database is going to get harder to access. So really everyone should have a copy of all important PII. :P

If something is on the cloud does 3-2-1 backup stop applying?

I would also like to know the answer. Would it be a good idea for the company to keep _encrypted_ backups on their machines/HDDs? Not a laptop somewhere, but something just a bit more involved.

i think for company-critical databases, the best you can do without invoking a terrible headache for your security officer is going multi-cloud: one big tech cloud, and one smaller firm that is completely disconnected from the other one

maybe they could even use a relatively inexpensive colo/baremetal provider to simply mirror the bigtech deployment on a smaller scale (would need to be quite flexible/vendor-agnostic to make that work...)

It would make sense to keep backup on hard drive stored in safe in office. Doing it weekly would be reasonable but would have to accept that going to lose a week's worth of data.

The main problem is that would outgrow single hard drive so would need NAS. Also, the transfer speed could be an issue as database gets bigger. Even if don't store all customer data, it does make sense to store all the configuration, keys, and secrets.

Yes. Having a copy you can "touch" is important. At the absolute minimum you should have it on another cloud service.

You can still do off-site backup to another cloud.

Only if you're willing to stake your company's digital existence on the reliability of another company's cloud service.

If anything, it increases the need for 3-2-1 backups: the original copy of all of your files are on somebody else's computer that you have no control over. Hopefully they're keeping it backed up, and hopefully they don't go belly up and pull the plug all of a sudden. So you can use a primary backup in another cloud service from another company that hopefully won't kill their product at the same time as the other one (again, you have very little knowledge or control of the way they run their data center). Ultimately, it's a good idea to have a copy of your data that you have control over, maybe in a big drive (or set of drives, tapes, etc) in the safe, rotated daily/weekly/however long your company can cope with losing in a major SHTF situation.

Excessive? Maybe. For what it's worth my shop is locally hosted with both local and cloud backups. I have never regretted having at least one backup of anything and it's saved my bacon (or my coworkers', boss', etc.) a number of times. I've been fortunate to never need to rely on a secondary backup, but I sure wouldn't bet the company on it.

Ah that makes more sense, I can't read. I thought that the project stopped working all together, hence the startup was finished. I didn't realize it meant that they simply lost enough customers to go under.

A company's source code is mostly valueless. A company's customer data is priceless.

As Fred Brooks said in Mythical Man-Month: Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.

Crazy, but even AWS resources are not unlimited. In 2023, I experienced multiple days where g4dn instances were not available in us-east-1 (in any AZ).

Heh, they still live in the real world where they have to order or build servers put them in a rack, configure them to be added to the pool.

At their scale lots of their stuff is custom and needs to be ordered at least 18 months in advance.

The fact that they can do capacity planning 2/3 years in advance and have very limited misses in a way that people are astonished that they have capacity misses is a testament to how good they ate at it.

I think a lot of people who have never really used anything other than a major cloud provider don't really understand what goes on behind the scenes. I was a sysadmin at a hosting provider and even though we were magnitudes smaller than AWS, we still saw a lot of the same issues, just on a smaller scale.

Yeah, for sure. I've got a couple team members that I brought along from cloud only, (one having never run any unmanaged k8s even) to building and maintaining on prem compute clusters.

'We had a lot of discussions along the lines of, 'Yeah, that's a great idea, but we can't just autoscale out of bad config or planning' or 'We can't just reboot a host and get a new one, we have to plan to take care of the ones we have'

It's been pretty fun

Were they paying for said spot instances? If so, that's mostly on AWS, not the client. (Unless AWS explicitly says in its TOU that instances can be taken away instantly due to "availability" issues. Which IMHO would be a suicidal policy for AWS to have.)

It is not suicidal and it is in fact what it says. You get 120s notice and there’s no capacity sla even for on-demand or so called "reserved instances". You need to setup https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-capa... to get a guarantee and they will obviously bill you for it even if you're not using it.

That's like a gamma ray burst hitting the planet and just vaporizing it. So unlucky. But obviously fate had other ideas.

Not really with spot instances it was just waiting to happen on a day with more demand than usual. That is 100% expected

More like a balancing rock statue in an area that can have earthquakes

But hey, think of the money they saved before that!

Having all your customers' data on instances local drives (sounds like they didn't use EBS) with no backups sounds pretty dumb, spot instances or not. Those weren't serious people.

Common mode failure.

Reply on news.ycombinator.com