O2 outage: 31m mobile customers unable to get online | Hacker News Reader

O2 outage: 31m mobile customers unable to get online

theguardian.com

62 pointsViperus23 comments

I'm seeing references to something Ericsson contributes to their network. It would very interesting to see what went wrong and in particular, what caused the change or deployment to be "un-revertable". My money is on a cascade of issues, some of which were latent and only triggered today.

This is, by far, the worst cell outage I've seen in the UK. There is zero connectivity via O2 in e.g. London, data or otherwise.

[1] https://www.standard.co.uk/news/world/o2-network-down-ericss...

Due to an expired TLS cert: https://www.ericsson.com/en/press-releases/2018/12/update-on...

I liked the "limited number of customers" yes that's true, it affect 2-3 companies, it's just that a single one of those companies had 32M customers.

Ouch.

Something that simple, given that full service still hasn't been restored?

There was worse on 7/7/2005 in London due to the bombings. Nearly all the providers fell flat due to overload including land lines and mobile phones. Also public transport was hosed. I walked 11 miles home in the end.

FM radio was about the only thing working. Thank you cheap nokia!

I have an iPhone now which is devoid of an FM radio but carry a multiband VHF transceiver with me most of the time now as that can pick up FM and do comms on 2m (I have an amateur radio license as well).

any links on how to get started in the hobby easily?

If you're in the UK: https://rsgb.org/main/get-started-in-amateur-radio/

If you're in the US: http://www.arrl.org/what-is-ham-radio

Anywhere else, check your local society. It's quite easy to get into and not particularly expensive. Less than $100 including a cheap handheld radio as a starting point.

https://status.o2.co.uk/

> We believe other mobile operators around the world are also affected.

Anyone else affected?

Softbank, Japan's 3rd largest mobile carrier having around 25% marketshare in the country, had a major outage today as well.

Ouch. Their status page seems to be overloaded, assuming that is the correct URL.

Why do so many status pages seem to crash/incorrect report when the service goes down?

Once you get on the page:

    We're really sorry but this system is currently
    experiencing some technical problems.
    Our engineers have been notified, so please check back
    later.

Because hosting your own status page seems like a fine idea until your service goes down and 1 million people come to figure out what's wrong or even worse, just lose the entire status page along with your infrastructure.

Best practice is to host the status page on completely separate infrastructure. This adds a lot of operational overhead often seen as disproportionately low benefit relative to the ongoing effort to maintain, until rare events like this actually happen.

> Best practice is to host the status page on completely separate infrastructure.

Perhaps they have - and it's overloaded.

Probably offset by the amount of people who can't access it due to their internet not working haha

I know SoftBank pretty well from an old job. They do not do ops very well is the minimum I can say on this.

Loaded eventually for me. Must be getting a huge surge in traffic.

Statistically speaking - yes. I realy want to know what O2 had in mind by saying that.

The issues are in some of Ericsson network software, and they have other customers apparently.

https://www.ericsson.com/en/press-releases/2018/12/update-on...

I thought initially it was a local thing like when Vodafone disappeared here because a falcon made a nest on the transmitter

University of Southampton?

Yup. 2006-2012, still live in Highfield.

I haven't had mobile internet since yesterday evening on O2, and I travel to London for work.

Luckily, there's WiFi at home, at work, and on the trains, so it's not really affecting me too much, but I'd really like to see the postmortem of such a huge outage.

Reply on news.ycombinator.com