Severed fibre cables disrupted internet access in eastern Europe, Iran, Turkey
bbc.com
bbc.com
https://saigoneer.com/saigon-technology/11885-sharks,-anchor...
https://tuoitrenews.vn/news/society/20190528/vietnams-intern...
https://tuoitrenews.vn/news/society/20171016/vietnam-grapple...
https://tuoitrenews.vn/news/city-diary/20170125/save-vietnam...
https://www.independent.co.uk/life-style/gadgets-and-tech/ne...
See https://en.wikipedia.org/wiki/Submarine_communications_cable
—
Come on, hurry up with that shit, huh?
What's this, more shit?
No, that ain't shit. That's a German lens.
I've got two shipments of those.
That's good stuff.
Can't use it.
What do ya mean ya can't use it?
That's a telescopic one.
That's the best one they make.
It's not German, it's Jap.
This is an adaptor, it's not a lens.
You bought two shipments
of Jap adaptors, not lenses.
Jap adaptors.
"Normal access to international sites, including Google, Facebook and YouTube, will be ensured, the Viettel representative stated."
https://tuoitrenews.vn/news/society/20191224/issues-in-3-sub...
It took a while to figure out what happened, and internet was out for a couple of days. Considering service was restored in 30 minutes in this case, we are getting a lot better at handling severed cables.
30 minutes?
This is absolutely mind-boggling to me. 30 minutes -- and you can identify exactly where the cut was, deep in the water, access it, fix it in the water? To me that sounds like a 30-month job not a 30-second job.
Anyone here with insight into this fix-process: could you please shed light on how this is done and how it can be so fast?
However, service is restored sooner than that by rerouting service over a different fiber cable or undamaged set of fibers.
Here is an animation describing the process https://m.youtube.com/watch?v=ZJblHOGXKo4
BGP itself has no real facility for dealing with circuit congestion.
So, I am willing to bet that dynamic routing was certainly in place, but the secondary/eBGP multipath destinations were full when the primary path went down.
And that necessitates manual action in instances where you aren't doing MPLS-TE, or don't have enough standby capacity, or both.
Physical repairs of broken submarine fiber takes weeks to a month.
Fiber tramsreceivers automatically negotiate the power needed to sustain the link. So in case of datacenter or city networker, you could do a split-of without introducing too much loss and error rates.
Trans-oceanic fibers are different beast but I guess someone figured it out.
https://en.wikipedia.org/wiki/Optical_time-domain_reflectome...
But then you still have to go there and fix it. I can't even get a pizza delivered in 30 minutes so I suspect the fix did not actually repair any cables.
Sadly some international ISP’s don’t like to share (even sanitized) routing tables...
I was in China at the time. All internet traffic was going the other way around the globe and was extremely slow or most often just timed out.
It took weeks to get normal internet speeds again.
It's routine activity for superpowers to tap undersea cables.
Why would surface ships be around, blatantly giving away what is happening? The likeliest answer is that they screwed up with their anchoring procedures.
[1] The best known being Blind Man's Bluff. See also: https://en.wikipedia.org/wiki/Operation_Ivy_Bells
Ah: https://www.theguardian.com/world/2011/apr/06/georgian-woman...
She was in Georgia.
Another super powerful plane now lays down a one ton hook on a big ass chain and drags it around the forest until it settles on something strong enough to stop it. It does not make the coincidence that big, especially when you consider that many cables land close to commercial routes.
(The surveying involved in route planning prevents dangling the cable in nonworkable ways)
But mostly yes. It is interesting to read a bit on submarine cables that our infrastructure depends that much on.
It is for instance interesting to know that their landing points are considered strategical locations and that at least US and Russian navies have trained at eavesdropping on them using submarines. Just throwing that in case you are wondering why you don't hear much about X-37's ability to rendez-vous and "repair" satellites.
of course, we were less than half joking.
"SSL added and removed here! :)"
But they're betting one day either a security vulnerability will be discovered, or computers will be fast enough to attack the encryption and allow them to read the data. So even though it's unreadable today, it might be in 10 years.
According to the NSA technical director at the time (2013) the Utah datacenter has a capacity around 5 Zettabytes.
Considering around 100 million hard drives are shipped per quarter, that would be over a decade of production.
Still ridiculous for information that is worth less and less over time.
So I looked into that quote from the NSA director. What was actually said, apparently, was that the center was designed to hold up to 5ZB, not that it actually did. That seems to be a design based on some extremely optimistic assumptions about future drive density, power consumption, and cost. Needless to say, those assumptions were a bit silly at the time and have only seemed more so in retrospect.
P.S. It looks like TechCrunch came to almost exactly the same conclusions that I did, for very similar reasons. https://techcrunch.com/2013/07/24/the-nsas-massive-utah-data...
https://arstechnica.com/information-technology/2015/10/how-t...
""" For commonly used 1024-bit keys, it would take about a year and cost a "few hundred million dollars" to crack just one of the extremely large prime numbers that form the starting point of a Diffie-Hellman negotiation. But it turns out that only a few primes are commonly used, putting the price well within the NSA's $11 billion-per-year budget dedicated to "groundbreaking cryptanalytic capabilities." """
Shoosh, we're all supposed to engage in the charade that it isn't going on.
Even equipment that appears to have been cleared out is probably hiding secrets in flash. The vendor of the equipment in this case had a separate command to wipe file contents. Deleting files just unlinked them in the flash fs.
Pretty much all of them (with the exception of the VPN PSKs) were sufficiently "generic" enough that I was convinced that they weren't device-specific, i.e., they were probably shared across many such devices.
According to the login banner, the firewall came from a casino.
I'm certain that my experience was not a unique one.
I mean, I'd kind of expect not to be allowed in everywhere in such a facility for many reasons. You can't go anywhere you want in all kinds of public infrastructure (water treatment plants, power plants, etc.).
Apparently their immediate response was just to reroute everything through the cables on the other side of the country that were still top, and let their bosses know it was happening. They didn't slow down to ask for permission to light up half a country's worth of bandwidth, then just did it.
I'd be pretty surprised to learn that was a "manual" decision made by humans.
I would expect -- especially in Google's case -- that the re-routing happened automatically (and nearly instantaneously, ~50ms) and the NOC engineers were simply "notified" about it.
Corporate speak for "obvious sabotage"...
"Suspected Internet Cable Spy Ship Operating In Americas For Over A Month"
https://www.forbes.com/sites/hisutton/2019/12/01/suspected-i...
I wonder if this was detectable undersea fibre damage, or just Iran/Turkey switching off multiple shore stations at once?
It seems like a no-brainer to use space as a backbone haul for inter-continent traffic vs. physical cables.
It might be slightly faster, it'll probably be cheaper, you can scale the bandwidth by deploying more (very cheap) satellites.
Most of all, the traffic cannot be cut, spied on, you can spread access points in multiple locations (i.e. resiliency via distribution vs. single point of failure).
Spying on RF signals to and from satellites was a major Cold War activity. Governments spent a lot of money on it and got really good at it.
You might say that satellite data is still safer than terrestrial cables because a particular adversary might not be present in the footprint of a particular satellite, but (1) they might be, because the equipment needed to intercept the downlink may be easy enough to conceal, and (2) satellite signals get intercepted in both directions, uplink and downlink.
The one countersurveillance benefit that I can think of is in bypassing cable landing regulations that are used to require cable operators to give governments access to the data on their cables. While governments may also try to place licensing regulations on the use of satellite data services within their territory, it's a lot easier for individual end-users to evade or ignore those regulations than it would be for them to surreptitiously land a fiber-optic cable! However, governments will also have an easy time monitoring downlink signals in their territory so they may not lose much access compared to the cable landing license approach, depending on how the operators handle link encryption.
Why? Wireless signals propagate faster linearly than fiber signals, no?
I don't have time to do napkin math, but if 2x orbital height + curvature path multiple < linear speed difference, wireless comes out ahead.
From memory, I think this held true for transoceanic distances.
Starlink is the other “not here yet” services are going to use low earth orbits for something like a 25-35ms latency to the satellite.
Assuming light travels at 2c/3 in fiber, it would still take less time for a signal to go around the equator back to the same spot than for it to arrive anywhere on earth if using a satellite link.
Spacex's satellites are a game-changer because they sit in low orbit.
https://news.ycombinator.com/item?id=21324653
"Starlink has the potential to provide lower latency around the globe than terrestrial fiber because light travels faster in vacuum than in optical silica fiber.
"Delay is Not an Option: Low Latency Routing in Space"
...
"We conclude that a network built in this manner can provide lower latency communications than any possible terrestrial optical fiber network for communications over distances greater than about 3000 km.""
This is only talking about latency. Bandwidth is a different game. Sure, any given fiber link will probably have better bandwidth than any satellite link. Cause the medium in between is much more consistent, leading to less signal degradation. However, fiber needs to be placed, connected, maintained. Moreover, fiber needs a physical space.
Meanwhile wireless just needs access-points at both ends, well-aimed antennas, and little interference in between. Notably, I believe star-link will use laser communication for inter-satelite communication, which essentially means perfect directional antennas. This stuff might be able to scale faster. It can certainly scale further because you don't need the space for all those cables.
And for small users much more likely to be affordable for a CPE terminal and monthly service than the smallest service options available via o3b.
It will not be a viable option to replace 80 x 100 Gbps DWDM wavelengths on submarine fiber cables.
https://www.cnbc.com/2019/12/19/federal-reserve-bank-glitch-...
My best guess:
a) The up and coming dictator Erdogan flexes his muscles for no particular reason. He has now probably demonstrated that he can cut two fibers at the same time using his all mighty turkish subs.
b) The Fed botches a software upgrade in the US
And this happens to happen at approximately the same time.
and as the Wiki entry says
>According to Beckert, cable cuts happen "on average once every three days." He further noted that there are 25 large ships that do nothing but fix cable cuts and bends, and that such cuts are usually the result of cables rubbing against rocks on the sea floor.
>According to Global Marine Systems, "Undersea cable damage is hardly rare—indeed, more than 50 repair operations were mounted in the Atlantic alone last year". While a cut in a cable crossing the Atlantic has "no significant effect" due to the many alternate cables, only a handful of Internet cables serve the Middle East. These disruptions are only noticeable because of the small number of cables
Huh. How much current is there down at the bottom of the ocean? I was imagining a mostly static environment.
Here is a figure for the waters around Antarctica though:
https://www.nationalgeographic.org/encyclopedia/ocean-convey...
>An estimated 35 million to 45 million cubic meters (between 1.2 billion and 1.6 billion cubic feet) of water per second are continually moved from the ocean bottom to the surface.
That's over a huge area though and still no real idea of speed.
> The company told internet service providers to connect to its other servers to "route around the problem".
I believe this means they changed their DNS settings and waited for it to propagate. This implies that the internet in general was accessible, but some larger companies that bought part of a fiber cable were inaccessible over that line.
Then in 2010 [2] when the volcano in Iceland grounded flights throughout parts of Europe we were similarly like "argh, sorry customers!"
Cut a few cables and knock out the 31 GPS satellites and the world would grind to a halt. It's terrifying how dependent we are on technology for virtually ever facet of our lives. Then we rely so much on air travel for delivering persons and freight, modern society is so incredibly fragile.
[1] https://en.wikipedia.org/wiki/2008_submarine_cable_disruptio...
https://en.wikipedia.org/wiki/Air_travel_disruption_after_th...
http://energyskeptic.com/2016/when-trucks-stop-running-so-do...
"When Trucks Stop Running, So Does Civilization"
and also
http://energyskeptic.com/2015/power-transformers-that-take-u...
"Large power transformers are essential critical infrastructure to the electric grid, and are huge, weighing up to 820,000 pounds. If large power transformers are destroyed by a geomagnetic disturbance (GMD) electromagnetic pulse (EMP), cyber-attack, sabotage, severe weather, floods, or simply old age, parts or all of the electric grid could be down in a region for 6 months to 2 years. "
You need to have a meter of optical fibre with you. Then, if you get lost, you just bury it and then ask the backhoe driver who digs it up for a ride home.
"This is a picture of the Mackenzie Valley fibre optic cable. Since it was laid, it's been eaten at by wild animals, struck by lightning, and run over by construction workers."
The obvious loser is the severed cable owner.
In cases where multiple cables are cut simultaneously the simple answer was that it was probably done intentionally.
Reminds of 2015 in SF: ”At least 11 physical attacks on these cables have occurred in at least 10 Bay area cities”
I don’t know who these people are, but I bet they play dress up in a black mask and have convinced themselves that the ends will justify their means.
Or just one location where primary and backup systems went through the same pipe, system, whatever?
Telco land is full of redundancy that eventually converges into a single point of failure.