All Circuits are Busy Now: The 1990 AT&T Long Distance Network Collapse (1995)
users.csc.calpoly.edu
users.csc.calpoly.edu
DSC was our customer at Cygnus. They were interesting as a customer (tough requirements but they paid a lot for them). For example if they reported a bug and got a fix they diffed the binaries and looked at every difference to be sure that the change was a result of the fix, and nothing else (no, they didn’t want upgrades).
That sounds pretty diligent!
We required that all other customers upgrade at least once a year (or maybe 18 months? I don’t remember).
That is insane. I really feel like modern SLAs are only getting worse - so much so that most companies fudge them, and try their hardest to never declare any sort of outage.
This is also why the software itself was designed with so many internal defenses and what I would consider an “immune system”. I’ve never seen anything like it even on an aircraft control system. That is mentioned in passing in the brief article but is easily missed if you don’t know what it’s referring to.
Most of what is done on the Internet at, say, “layer 5 or above” isn’t at all important so there’s no need for this level of SLA, but the actual backbone carriers do still carry SLAs at around that level. With packet switching it’s easier for them to provide than it was in the days of the 4ESS and 5ESS.
I wasn't aware of erlang the unit (measuring telephone circuit load) and at first thought this had something to do with the language.
Calls per second X seconds per call = no unit.
Call_starts per second X call.seconds per call_start = calls as the unit
Networking is the last bastion of SLAs that actually seem to matter.
As a result, five or six nines availability is pretty standard. Six nines means you can lose no more than 31 seconds a year. As a result, traditional telco had all kinds of cool tech in it, eg the Nortel DMS had live code patching as early as 1990 iirc.
At one time I worked at a vendor of IP telco switches, aiming to replace the legacy Nortel and Lucent with smaller tech. We had to learn some very, very hard lessons about reliability, but we eventually got there.
Today, I see cloud hyperscalers claiming that can run telco workloads, but I remain pretty skeptical until they can prove that they can switch a call, mid stream, from one node to another, without losing the audio, while transcoding it from one codec to another. Im not saying that public cloud needs to make the same tech choices re resilience, but today's web tech absolutely will not cut it.
The thing is, traditional telco can't do it anymore either. CenturyLink (aka Lumen) has broken 911 at least twice since 2018 when I moved into their territory. I've got the county emergency dispatch center in my address book, cause chances are, they'll do it a couple more times.
(it might sound crazy these days to trust "someone on the Internet" but really: what's the incentive not to tell the truth about something like this?)
Nice to see that "should have used Rust" has been a thing since before Rust existed.
I briefly had an interest in learning Q, then looked at some code: https://github.com/KxSystems/cookbook/blob/master/start/buil...
Why not just build what you need with C/arrow/parquet?
The one I am talking about hasn’t been released yet!!
By having everything so standardized and consistent, they had the exact same failure mode everywhere and lost redundant fault tolerance. If they had different interoperable switches, running different software, the outage wouldn't have been absolute.
When large complex distributed systems grow organically over time, they tend to wind up with diversity. It usually takes a big centralized project focused on efficiency to destroy that property.
The practical downsides of this diversity live in the complexity of the interop (often slowing feature velocity), operations, and procurement/support.
But issues like the AT&T 4ESS outage have occurred before in IP networks, as an example, in some BGP bug. Diversity alleviates some of the global impact.
You can sometimes play this game with vendors because you want them to give you an interoperable interface so that you avoid vendor lock-in and have better pricing, but that’s a secondary benefit and staged rollouts should still be performed even if you have heterogenous software.
But when it inevitably does, you should be prepared for a full system simultaneous restart. Ie. So that no 'bad' signals or data from the old system can impact the new.
That is the sort of thing you should practice in the staging environment from time to time, just for when it might be needed. It could have taken this outage from many hours down to just many minutes.
Don't get yourself in that position.
‘The Day the Phones Stopped Ringing’ by Leonard Lee
Definitely going to look into this! The whole *ESS architecture still underpins a lot of the telephony system. There are quite a few still running, even though other TDM equipment is being phased out.
I wonder if peer review would have caught it.