Hacker News Reader

Comment by pen2l | Hacker News Reader

> Considering service was restored in 30 minutes in this case, we are getting a lot better at handling severed cables.

30 minutes?

This is absolutely mind-boggling to me. 30 minutes -- and you can identify exactly where the cut was, deep in the water, access it, fix it in the water? To me that sounds like a 30-month job not a 30-second job.

Anyone here with insight into this fix-process: could you please shed light on how this is done and how it can be so fast?

View on HN

No, it's a process of ISPs rerouting transport traffic. Onto other routes at OSI layer 1.

Physical repairs of broken submarine fiber takes weeks to a month.

Identifying the break can be done really quick:

https://en.wikipedia.org/wiki/Optical_time-domain_reflectome...

But then you still have to go there and fix it. I can't even get a pizza delivered in 30 minutes so I suspect the fix did not actually repair any cables.

It’s about a 1-3 month repair job to acquire replacement cable, load it into a ship, and sail out to the break point.

However, service is restored sooner than that by rerouting service over a different fiber cable or undamaged set of fibers.

The cable is not fully replaced. They lift the cable up to a boat and repair it (replacing the damaged bit and joining it to the original) and lower it back down.

Here is an animation describing the process https://m.youtube.com/watch?v=ZJblHOGXKo4

Amazing really that the optics at the repair site are still good enough to carry the bandwidth.

They don't replace the cable typically, they fuse it with epoxy and it usually works pretty efficiently, mostly taking the time scheduling the boat and equipment the repair is likely a couple hour job at most and 12-24 for the epoxy to dry.

No one fixed the cables, they have rerouted traffic through other links. Most of the traffic was automatically rerouted, that's how BGP works.

If it took 30 minutes that's not BGP, that's some people at ISPs manually provisioning new circuits on DWDM systems, using a moderate amount of automation tools, and bringing the circuits up facing routers at both ends. Or doing something like creating a few new VLANs in existing larger 100/200/400Gbps intercity transport links, and trunking those VLANs through to routers on each end. A topology change in BGP is a couple of seconds.

BGP may or may not still be involved.

BGP itself has no real facility for dealing with circuit congestion.

So, I am willing to bet that dynamic routing was certainly in place, but the secondary/eBGP multipath destinations were full when the primary path went down.

And that necessitates manual action in instances where you aren't doing MPLS-TE, or don't have enough standby capacity, or both.

Keeping sufficient hot standby submarine capacity live and in contract is very costly in $/month.

That's why I've specifically wrote "most", of course it's not all BGP, BGP routing algorithms do not need 30 minutes to change routes.

Do international peers tend to use BFD?

I understand your astonishment, but if they had properly configured dynamic routing, it would not have taken that long.

Sadly some international ISP’s don’t like to share (even sanitized) routing tables...

I'd guess traffic was rerouted.

I'd guess it was done on purpose, so that it be rerouted.

Rerouted to a backup, I'd guess?

My first guess would be re-routing traffic away from the damaged cable(s)

An event like this, where multiple fibers are disrupted and magically fixed less than an hour later just smells like the NSA, one of the other five eyes or a friendly telco installed a fiber optic beam splitter.

https://en.wikipedia.org/wiki/Beam_splitter

I don't have any traceroutes so this is speculation but the probably just route around it if it only takes 30 minutes. I was under the impression that a beam splitter can be installed without disruption. If they fucked up and broke the cable installing a beam splitter, a 30 minute fix is still pretty impressive.

I‘m by no means an expert but I did some datacenter work. I wonder how this would be possible to install a beam splitter without disruption? Don’t you need to plug the fibers out, put the splitter in and then plug the fibers back in the splitter? I was searching YouTube but couldn’t find a video.

They bend the cables, thus allowing part of it be tracked with some residue light coming out of the fibre.

I agree, this might be possible but I wonder how good the quality is then? A splitter will give you 50% light. That limits the length of the cable too, doesn’t it? How many errors will be there when you bend the cable and get maybe 15% light of encrypted traffic, and you can’t signal any retransmission?

On run of the mill singlemode fiber loss is roughly 0.5dB/km. Splitting off half of the light (and you don't need half, much less is fine), would result in 3dB of loss.

Fiber tramsreceivers automatically negotiate the power needed to sustain the link. So in case of datacenter or city networker, you could do a split-of without introducing too much loss and error rates.

Trans-oceanic fibers are different beast but I guess someone figured it out.

You don't need a beam splitter to listen in on traffic.

The light is kept inside the fiber by the change in refraction index in the radial direction of the fiber so there is a total internal reflection. But if you bend the fiber enough then reflection is not total and part of the beam goes outside.

Reply on news.ycombinator.com