Overheating datacenter stopped 2.5M bank transactions
theregister.com
theregister.com
> Upon the outage, both banks immediately activated IT disaster recovery and business continuity plans.
> "However," according to Tan, "both banks encountered technical issues which prevented them from fully recovering their affected systems at their respective backup datacenters – DBS due to a network misconfiguration and Citibank due to connectivity issues."
Perhaps regularly flipping between them on a regular basis, at least for a short time, so the 'passive' part of the active-passive pair gets some work, is worth considering.
"An untested recovery plan is worth less than the paper on which it is printed."
I'm sure there are many other versions of the quote
There’s a harness connected to a rail above you to catch you if you fall.
In the video the man gets halfway across and the cable attached to his harness just pops off. The instructor behind him instinctively reaches out as if he can use The Force to catch him if he biffs, but he makes it to the other side unharmed, and is all smiles until the cable catches up and bops him in the back. At which point he turns around and connects the dots.
It’s a bit like that. And this is the image that comes into my mind every time I discover we have untested recovery processes while we are in the middle of a recovery.
Maybe I'm just too skeptical.
On paper, there are plans and redundancies upon redundancies, and everything is perfectly safe.
In practice, I haven't seen them exercised, and some of these plans sound incredibly optimistic...
If you’re going to use the word strategy this way it has no meaning, which kills the conversation and communication in general.
I think you’re confusing strategy and behavior. Strategy is a plan to do better than default behavior.
Writing up a plan for disaster recovery is a strategy for disaster recovery. The pros are that it is cheap and ticks the regulatory checkboxes. The downside is that it has a high probability of a high recovery time in the case of a disaster.
My point is that there are different pros and cons to every strategy that you can pick. I also don't think that all strategies are better than "default behaviour" whatever that is.
For meeting dogs of course.
The problem I have is that it seems like you agree with other people in this thread, but real world experience tells me that stating things this way is more fuel for the lazy and for anarchists than it is support for the SREs.
Don’t let the idiots think for a moment they are entitled to use a word as heavy as strategy. It ends farcically.
Writing it down doesn’t make it a strategy. Satire is also written down.
An analog is when I was in the military we wore every level of MOP gear once a year to ensure that your gear functioned correctly and that you knew how to wear it. On a second day we would enter building filled with chemical irritant where you must dawn your mask, clear it, breath, and communicate. All of this is part of proving the plan which equivocates to strategy. We've not faced an imminent CBRN threat in decades, but we still do it because we maintain a strategy for dealing with it.
In this case, if you have a plan for how to deal with too much heat or too little power but you don't actively put the systems that transfer the responsibility or get put under stress during these disaster events then you never actually know for certain that they'll work. Therefore, they are not strategies, just plans.
It’s two steps away from Apollo 13. Build a CO2 scrubber out of parts in this box. You have twelve hours until the crew suffocates and dies.
True. But getting the PHB's at a bank (and perhaps the banking regulators) to sign off on doing that...could prove to be a non-trivial task.
Outages happen.
Could lead to more multi-cloud adoption among financial services companies who are in scope.
(no relation to the other DORA, DevOps Research and Assessment, by the way)
Cloudflare. Equinix. No accountability.
Have a problem with a service and you find out the company you bought from subcontracts to an LLC you've never heard of which in turn subcontracts to four other LLCs one of which is actually liable but only if you can figure out which one, and oh by the way yours has two employees and it's headquarters is a UPS Store box, good fucking luck.
Corporations (and other legal entities used for business) are legal fictions that allow people to pool capital for productive endeavors and yes, with limited liability on the parts of the owners, because we as a society decided that encouraging productive endeavors this way is better than the alternative.
In a world with no limited liability, you could happily wring the owner's neck if they screwed up, but I'd imagine society as a whole would be a lot poorer off, because very few people would take that risk to engage in productive endeavors.
> Have a problem with a service and you find out the company you bought from subcontracts to an LLC you've never heard of which in turn subcontracts to four other LLCs one of which is actually liable
It doesn't matter, because the company you actually paid money to is the one that's responsible to you.
Oh, people would make risky ventures alright, but you’d see a lot more private security of the Pinkerton variety.
I'd say more than a few people would like that opportunity with Boeing
Worth noting that legal 'collectives' date back to Ancient Rome, city charters, monasteries, guilds in the Middle Ages, etc. While they are currently called corporation, from the Latin corpus for "body (of people)":
* https://en.wikipedia.org/wiki/Corporation#History
The original term was from the Latin universitas, and were universities in (relatively) more modern times when the concept saw uptick in usage (1100s AD).
* https://en.wikipedia.org/wiki/University#History
The first collectives were either groups of students coming together to hire teachers (Bologna), or teachers coming together (Paris).
* https://en.wikipedia.org/wiki/University_of_Bologna#History
* https://en.wikipedia.org/wiki/University_of_Paris#History
Your local municipality, county, country, etc, are all legal fictions that act as a legal person. Turns out the concept is handy in many situations:
It's shitty landlord syndrome all over again. Tenants should have the right to complain and wave their sharpened pitchforks at them.
Literally in this case.
After 9/11, the "zero nines" whitepaper was released to push for cloud (never let a disaster go to waste).
I have hands-on deployed many K8s+Ceph clusters. Ceph's block storage can be deployed multi-AZ where multiple datacenters each with their own power systems are in close proximity. There are methods like Raft (used by K8s and Patroni) and a really great method demonstrated by Google's CloudSQL that uses a SQL table to infer which SQL server should be a "leader".
So like wtf guys.
One of the banks, DBS, has had several outages in recent years, but this seems like sloppy writing.
(I guess over time those should converge, but sometimes takes a while or doesn’t)
Power is the killer, and it's doubly bad. Not only is it expensive, you're using a lot of power to keep things cool given ambient temp fluctuates typically between 24-33C, 60%-90%Rh 24/7.
There was a moratorium on new DC builds in Singapore, so there's also a huge demand/premium on rack space in SG.
And that's just 2 nodes out of the current 15K-20K.
[1] i.e. Bitcoin second layer.
They don't? Then surely you wouldn't mind sending me a few bitcoins over a lightning channel? Pretty please?
> no one needs to send 1000 transactions of single penny
There is no such "single penny" limit.
> then have the exact same problems as everyone else
What problems?
I could write down a seed phrase on a piece of paper and hand it to you, and that's a completely offline transaction.
It prevents the parties of a transaction from double-spending balances behind each other's backs.
It also doesn't require trusting the other party once a transaction has (almost instantaneously) completed.
The fact that I even have to ask you already demonstrates how shitty your third layer is compared to Lightning, not to mention all the other obvious ways in which it's infinitely worse.
In any case, seeing that I don't have a valid signed BTC transaction from you (because your 3rd layer suuucks), you don't want to allow free exchange between your layer and the upper layers, and I wouldn't want to bother with court litigation even if you were serious, well, have a nice day and good luck with your third layer lol
Except that those transactions don't count until they get on the main chain
While you quoted me as:
> those transactions don't count
You lost some of my message. Maybe your are reading it through a lightning node and it went offline half way through.
There is no such "single penny" limit.
No one said that, but lightning's examples are always these nonsense situations of paying a single party a thousand times.
What problems?
A transaction rate that could fit over the bandwidth of a 28.8 modem.
If you have to misquote people to make your point, maybe you're on the wrong side of the argument.
So what's your point? Your point is that the balance will be reflected in the first layer after some time instead of being reflected after each and every lightning transaction?
What does that have to do with what we were talking about? The point of this thread was how many transactions per time period you could do with Bitcoin, not whether the balance will be public information in real time?
> lightning's examples are always these nonsense situations of paying a single party a thousand times.
It's not a nonsense situation. Micropayments is something you can (trustlessly) do with Lightning but not with traditional payment technologies.
It doesn't mean you can only do micropayments with Lightning.
> A transaction rate that could fit over the bandwidth of a 28.8 modem.
I just told you that Lightning can transact orders of magnitude more than that.
You can do millions of Lightning transactions per Bitcoin transaction.
You are complaining about Bitcoin's transaction rate limit but my whole point is that Lightning allows you to have an effectively limitless transaction rate (under certain conditions, at least), even if it requires occasional Bitcoin transactions to settle balances.
Transactions that don't count and aren't useful to anyone until they become a bitcoin transaction, and that's still expensive and slow.
No one cares about it because it is terrible technology that doesn't actually solve anyone's problems, because the people that made it made the problems in the first place.
I have no idea why you're making such an obviously false assertion.
These transactions obviously "count" and are useful to the entities running those 15K Lightning nodes (many on behalf of multiple customers), which can all transact amongst themselves over the network, even without touching the Bitcoin network (or occasionally doing it just to settle balances).
These transactions have almost exactly the same cryptographic and trustless assurances as normal bitcoin transactions, which are far superior assurances than those provided by traditional payment / money transfer technologies.
> No one cares about it because it is terrible technology that doesn't actually solve anyone's problems
I think you are confusing your own ideas and problems with other people's ideas and problems.
Just because you don't care about it doesn't mean that other people don't (or that even you won't in the future, if your circumstances change).
I can trade IOUs with a bunch of people in my local cult, but they aren't money until someone makes an actual USD transaction.
Lightning transactions are not based on IOUs.
IOUs can be defaulted upon, while BTC received over Lightning cannot.
Also, you need the cooperation of the other party to redeem IOUs. Lightning transactions don't require cooperation for settlement, they can be settled at any time by any of the parties.
All of this means that the BTC that is exchanged over Lightning is just as valuable as any other BTC, unlike IOUs.
> they aren't money until someone makes an actual USD transaction
That's because your definition of "money" is USD, so your sentence is a meaningless tautology.
It's like saying "raw beans aren't real food until someone cooks them". Uhm, sure, but does that mean raw beans aren't valuable or that you can't easily convert them into food if necessary?
And what's your point even? This point is equally applicable to the first layer.
BTC received over Lightning can easily be exchanged for USD if you want, or (even easier) for BTC in the first layer, so why does this point matter at all?
No, btc is money too, lightning transactions are useless until you sync them with the main network. Skip the middle man that no one wants. It has been 10 years, no one wants this, it doesn't solve anything. Every other currency has bigger blocks and never had this problem in the first place.
This is absolutely false. Why do you even think this??
Heck, have you never seen people buying real world stuff with Lightning transactions in El Salvador?
Are you aware that if you receive BTC over lightning, you can spend them over lightning, with a different party, without hitting the main network?
> Skip the middle man that no one wants.
What middle man? Is the middle man in the room with us?
> no one wants this, it doesn't solve anything
False and false.
15K entities want this (at least, i.e. not counting hosted wallets). And it solves the limited transaction rate, which you were complaining about.
> Every other currency has bigger blocks and never had this problem in the first place.
And when they get moderately popular, the size of their block history becomes prohibitively expensive to download and verify.
All while not really fundamentally solving the problem, but rather only allowing a slightly faster transaction rate which would still not be able to compete with VISA (as an example).
The middle man is a layer that no one wants and it is not in the room with us now because no one wants it.
15k "Entities" means a bunch of VMs spun on the same servers.
And when they get moderately popular, the size of their block history becomes prohibitively expensive to download and verify.
This is where the technical rubber meets the road with all the lightning sycophants. This is insanely untrue and it never has been. No one is ever able to prove this with numbers, they get out the playbook of dealing with a claim they can't back up.
Bitcoin right now has the bandwidth of a 28.8 modem. At one point single transactions cost more than the space to store the entire chain. Normal people don't even need to sync with the main chain if they don't want to. The actual numbers of bandwidth and storage of this argument are so far removed from reality I can't believe people still say stuff like this. It's pure propaganda.
You're only considering the bandwidth to monitor/receive transaction confirmations in real-time and not the bandwidth to download the transaction history, which cumulatively grows over time, forever.
It's already at more than half a terabyte, which you certainly can't download over a 28.8 modem.
> At one point single transactions cost more than the space to store the entire chain.
The problem has never been about space, it has always been about bandwidth.
> Normal people don't even need to sync with the main chain if they don't want to.
Sure, but you need to put some bounds on how fast the main chain can grow to ensure that anyone (even the people, say, in El Salvador or the Central African Republic, both poor countries where Bitcoin is legal tender) can verify it if they want to.
> The actual numbers of bandwidth and storage of this argument are so far removed from reality I can't believe people still say stuff like this. It's pure propaganda.
Or perhaps you are privileged and you don't even notice, and therefore you are incapable of empathizing with others who are less privileged than you (in this specific instance)?
In any case, even if you were to ignore this, increasing the block size even by a factor of 100 would not fundamentally solve the problem.
It would still be a pathetically low transaction rate compared to Lightning and it would also certainly constrain Bitcoin usage as it was intended (as a trustless currency/asset).
That's not to say I would be against increasing the block size by a reasonably small factor, but I'm certainly not in a position to judge whether this should be done right now. I think it'd be better to ask the people in those countries, or even poorer ones who might one day make Bitcoin legal tender as well...
Even though you already said it's all about bandwidth, next you'll say it's about the CPU to verify, even though that takes a second per block at most.
Then you'll say it's actually about hard drive space, even though a 512 GB SSD costs $24 dollars while fees for a single transaction spiked above $30 on average earlier this year. Right now they are at $6 USD!
https://bitinfocharts.com/comparison/bitcoin-transactionfees...
You didn't make a technical argument with numbers because you can't. Your next reply won't have real numbers either, it will be more attempts to obscure what you can't defend with attacks, distractions and more copy and pasted arguments that don't add up.
None of what you just said makes any sense.
BTC has been consuming all of the planets energy since at least 2020[1]. This increased transaction rate of Lightning just means that BTC will soon be consuming over twice the total energy produced by humanity, which is unsustainable.
Of course, people who don't understand the scientificness of this all are still defending bitcoin as if we don't already have enough payment apps.
[1] https://www.weforum.org/agenda/2017/12/bitcoin-consume-more-power-than-world-2020/That is plainly false[2], and the linked article does not say what your sentence purports it to say. :D
2: as evidence I can point out that I have seen at least one car moving since 2020. That was clearly using energy BTC did not consume. Same with observing at least one light bulb illuminating, or at least one person breathing.
> Answer: Air conditioning. Air conditioning was a most important invention for us, perhaps one of the signal inventions of history. It changed the nature of civilization by making development possible in the tropics.
https://www.vox.com/2015/3/23/8278085/singapore-lee-kuan-yew...