Hacker News Reader

Comment by yabones | Hacker News Reader

I wonder if it will be the same cause as the big Rogers outage in '22, a good old fashioned BGP botch.

https://www.reuters.com/business/media-telecom/rogers-commun...

https://en.wikipedia.org/wiki/2022_Rogers_Communications_out...

Diff with Rogers is that they took out their entire network: cellular, home/biz internet, home phone, corporate circuits (including MPLS links), most cable TV, a bunch of their broadcast radio (AM/FM) network just dead dead dead.

Well, their towers were sorta up (as they couldn’t remotely turn them off since the network was down), so if you had a Rogers SIM, a call to 9-1-1 wouldn’t failover to other networks because the device made just enough of a handshake to try and fail on the Rogers network. A flaw in GSM I reckon.

Apparently the workaround was to remove/disable your SIM and hope another network has a stronger signal.

Oh, and the CTO was on holiday and had no idea for a while because… their phone was on roaming with Rogers and therefore dead.

I wonder if Rogers still does planned-in-advance multi-stage potentially-enterprise-breaking updates on Fridays

In a financial company I worked at we would do some of the biggest, riskiest changes at 5pm on a Friday (or Saturday evening if we were worried about impacting international trades). The logic being that we would have the most time to fix things before markets open monday.

Our release window is Saturday morning. All the support people are on, most users are not, gives us 36hrs. We absolutely do not release during week if we can help it. But we are traditional ERP application so pretty much everything we do is contrary to the HN/modern zeitgeist :-)

Being able to release safely during the week is super important for eg financial services, for fairly obvious reasons.

In trading and market making contexts for instance, we release 100s of times a day — including Fridays. This includes bog-standard infra changes like roleswaps and server rebuilds. The releases that happen on weekends tend to be highly disruptive infra changes, eg unexpected changes to some kind of physical connectivity where we’re not comfortable with carrying weekday risk.

We didn’t explicitly set to to optimise QoL for engineers (the real driver for safe intraday change was being responsive as a business) but not usually being on call on weekends was a big plus.

> All the support people are on

Either Rogers thinks it can’t make mistakes, or nobody was informed that a potential enterprise breaking change was taking place.

Took some hours to get any official indication from them (partly because of their employees dependency on the service they provide).

Yes. It is either Friday evening or Sunday in finance.

Well this was like 6AM Friday Eastern time, soooo…

I know one outcome of it was to ensure that they were equipped with SIMs for Competitor networks just in case

You're saying that Rogers personnel now have non-Rogers SIM cards?

I mean, incredibly critical personnel probably should be! There may only be a few dozen such people, but I wouldn't want the added chaos caused in the event of a Rogers outage if I couldn't get in touch with the key decision makers and most critical operations engineers because of the very outage they're meant to fix. And in the e-sim era that is hopefully very cheap and without any real downsides.

I’m surprised the CTO on holiday didn’t have a sat phone.

I'm in the industry, though not a CTO. Nobody [that I know, on the tech side] has a sat phone.

The '22 Rogers outage, hah. As I recall it didn't affect me at all since I was at home and work in Vancouver all day… but it was a great excuse for not responding to workplace on-call messages which I got in the evening

> Well, their towers were sorta up (as they couldn’t remotely turn them off since the network was down), so if you had a Rogers SIM, a call to 9-1-1 wouldn’t failover to other networks because the device made just enough of a handshake to try and fail on the Rogers network. A flaw in GSM I reckon.

Didn't know that part, amazing.

It sounds kind of like connecting to a WiFi access point which has a broken/non-working uplink to the Internet. Modern smartphones pretty much automatically detect and avoid such APs, and indeed the whole SSID if they need to, but it sounds like the stuck-in-1985 2G baseband layer has no equivalent connectivity check.

> their phone was on roaming with Rogers and therefore dead

I thought your phone uses all available networks (ie the strongest one) while roaming. Is that not the case?

When roaming, your home network is needed for routing incoming calls to you, and handling authenticating your device to the visited network.

And Australia 2023 outage of Optus, also BGP related: https://en.wikipedia.org/wiki/2023_Optus_outage

Reply on news.ycombinator.com