Did a Chinese hack kill Nortel? (2020) | Hacker News Reader

Did a Chinese hack kill Nortel? (2020)

bnnbloomberg.ca

162 pointsmepian84 comments

Nortel execs killed it but there was rampant theft. I remember working on the WiMAX program and one weekend someone had downloaded years of research (Shanghai I think ) circa 2004/5 a few angry emails but remember everyone wanted the China market , outsourcing was cool etc. They brought in Motorola exec and GE exec with a background in washing machines to work in enterprise ( I cannot stress how different a line assembly is to multi service switching ). Toward the end they attempted to get a bail out but Oshawa region won out ( US car companies employing more people and more votes meant more at stake ). I joined in 1997 and left around 2008 just before the collapse. Wireless was sold to Ericsson and I heard they recouped costs in about 9 months , Nokia picked up the passports ( wan switching ) and dropped their own products … tons of stories like this. Interesting side story I joined the brand new content networking division around 2000/2001 worked on a new L4 to L7 switch ( TCP splicing and load balancing and higher ) that was developing a new network processor around Lexra cores at 10G but it all fizzled .. I do remember thinking what an absolute beast this chip would be “candlestick” would be. The cisco guys whom nearly took over had it right , down size to core networking … half staff ( it was around 40k when I started and 130k when I left ). China certainly did nefarious things but it was greed and incompetence in leadership that killed it.

Also, Mike Zafirovski (Nortel CEO) asked for financial support from Harper, who refused. Nortel had many problems, but Stephen Harper (prime minister) and his Conservatives killed Nortel and the telecom industry in Canada with this decision. Now the same people will blame China. That was the moment that ended the business.

Bailing out the private sector every time they go bust due to incompetence won’t solve the problem (in this case, problem of management and security incompetence).

If they were bailed, what would've changed? Even car manufacturers weren't disfunctional to the point of not being able to turn a profit on a line of product that when sold to another company took only 9 months to be profitable?

Not only that, but from what I'm understanding it seems like the entire company was compromised so deeply that almost every email/document seems to have found its way back to China. And it took years after their bankruptcy to finally get an accurate idea of how wide the breach was because Nortel itself seemed to really not care about that at all. Why then would the Canadian government bail out and finance Nortel when it amounted to bankrolling Chinese research?

I think the real mistake was to let the expertise go to complete waste, it's just mind-boggling that only Huawei was smart enough to take advantage of such a concentration of talent. The Canadian government could've let Nortel die but financed a successor to replace them, or done something to help out the engineers left in the dust

But I guess Canada has a very long history of losing every single cutting edge tech hubs it has, while doing very little about it. Because why even bother when you are a petrostate?

I grew up in the Ottawa area. I remember watching so many friends' and classmates' parents suddenly go from successful to unemployed. What's worse, their pension plan was mostly invested in Nortel stock. They lost their jobs and their retirement. Most of these people were in their late 40s already.

There weren't other companies to hire them all. Ottawa's tech scene was mostly Nortel. Without it, what could they all do with the skillsets they had? One close friend of mine, his dad became a cabinet maker. Just started over entirely.

Was it Huawei? Hard to say. But whoever hacked Nortel attacked my community, in my mind.

+1. I interned for them in '99. Everyone was talking about the stock and buying new houses. A few years later I had to get a letter from my previous manager for immigration purposes. It was difficult finding anybody from my previous team within the company.

Pretty sure the absolutely rotten executive team didn't help either. https://en.wikipedia.org/wiki/Nortel#Criticism_and_controver...

I lost my job and pretty much my entire investment portfolio when this happened. It was devastating. I went from doing well, to (by the end of the time I was out of work) not having the money to both pay the next month's rent and buy food. Luckily, my family has since bounced back and we're in a great place now. But it taught me a solid lesson about not having most of my eggs in one basket.

I moved to Ottawa in 2000 to one of the few defense companies there. The boom was happening and folks were being given 10-15% pay rises, just to try and stop the bleeding to Nortel, as they were paying huge salaries for software engineers back then. Needless to say, when Nortel imploded, we were hiring for a new development program (C++, VxWorks, real-time, DO178B etc etc)and I interviewed dozens of ex-Nortel folks, of all kinds of levels and skill-sets. Now, telecoms isn't airborne software, but we found some absolute gems of engineers that were cast-aside by Nortel. Nortel's loss was our gain in some respects, and another upside was bringing their development best-practices and Nortel engineering experiences with them. My god, we even got to know their product structures after doing so may interviews, e.g. OC-192 architecture always seemed to be particularly good interview rabbit-hole.

I had a friend in uni whose dad was somewhere in R&D at Nortel. I really wish I got to know them better so I could know more Nortel lore.

> What's worse, their pension plan was mostly invested in Nortel stock.

Why? Didn't they see what happened with Enron? Or did they try to diversify but couldn't?

While we're on the topic, how many people keep their RSUs vs how many sell?

The safe advice, and what I do, is to just sell immediately and diversify. Why keep all your eggs in one basket?

Maybe it varies by company but at every place I’ve worked the company sells a percentage of the vested RSU’s to cover tax liability so you’re left with post-tax shares. In my mind I agree it’s better to sell since there’s no tax advantage to holding and really just lots of downside potential.

Plus holding large amounts of your net worth in your company’s stock where you can only sell it at certain times of years doesn’t seem like a great plan.

ISO’s are different of course. It could indeed make a lot of sense to not exercise them for tax purposes.

I think the biggest travesty was that the Canadian government was unwilling (or unable) to go after Huawei (and China) for the damages that this attack caused. At this point, USA has almost completely destroyed Huawei's global consumer electronics business in retaliation for similar attacks on American companies[0][1]. There is no more such thing as a Huawei mobile phone, at least in western civilization.

That is the kind of response that I would expect. But I never heard about anything even close to that from the Canadian government.

[0] https://www.justice.gov/usao-wdwa/pr/chinese-telecommunicati...

[1] https://www.justice.gov/opa/pr/chinese-telecommunications-co...

The United States has a lot more leverage than Canada on these sorts of things, and back when this all happened people were much less negative on China.

Canada could ban Huawei domestically, maybe. But a wider ban that would really hurt Huawei can only be pulled off by some later entity eg US, EU.

I don't know, I think it's possible Canada has more weight in this kind of thing than you'd expect from it's basic stats. It might not harm Huawei much directly with a ban, but I can easily see that kind of thing spilling over to countries like the US, UK, and Australia as long as they don't intentionally screw up the PR for it.

> There is no more such thing as a Huawei mobile phone, at least in western civilization.

They seem to be alive and kicking in Swedish shops though.

https://www.pricerunner.se/sp/huawei-smartphone.html

The newest phone I can find on that list was released over a year ago, so it's just old models (old stock even?)

>There is no more such thing as a Huawei mobile phone, at least in western civilization.

That is not true at all. Most countries outside the US have them, especially in "western civilization". Hardly anyone follow the US line these days. Family members just bought two of their newest.

Are Your family happy with Android-like OS without Google? Huawei smartphones are technically still sold in Japan without Google, but no one buy it, despite previously very popular.

(IMO Google Android ban is overkill)

“Was it Huawei? Hard to say. But whoever hacked Nortel attacked my community, in my mind.”

When the managerial class have finished abusing and ripping value from the company and its workers, it is a classic for them to lay the blame with the others: the inhuman Japanese, the dirty Mexicans, the cheating Chinese - they took your job and stole your wealth - not us, we are on your side.

Amazing how it still works and how little people learn from their own history.

The xenophobic blame game is always absurd. In this case, it doesn't even make basic chronological sense. Nortel collapsed before Huawei even established a major international presence. Huawei was barely even selling in the same markets as Nortel.

There were many reasons for Nortel's collapse: huge numbers of unprofitable acquisitions during the dot com bubble, the collapse of the bubble, the fall in demand for networking gear after the bubble, financial fraud. To pin the blame on a company that was barely even competing with Nortel at the time, and which just so happens to be the bogeyman in 2021, is just too convenient.

I don't know the details of Nortel, but I saw Ericsson post dot-com bubble from the inside, and that company was being stripped of anything of value, right down to cancelling cleaning and free fruits in the office over outsourcing everything to the cheapest bidder.

The rise of ZTE and Huawei comes much later (around the introduction of 4G - they start winning contracts in Scandinavia), and to me mostly because they were willing to invest in research at a time when western companies were not.

I'm not sure about that time line. I can remember BT replacing Marconi with Huawei backbone routers for its 21CN project, that was early 00s.

Huawei, from nowhere, suddenly had very cheap networking products. So cheap they effectively killed Marconi's bid.

It was an open joke at the time that they were copies of Cisco products (the other winning supplier) down to the bugs. No one cared because they were so cheap.

Personally, I blame BT's executives for not seeing the value in keeping a local manufacturer (just up the road) as one of their suppliers.

Huawei had almost no overseas revenue until 2004.[1] Nortel's collapse was well before that. By 2001, the were already headlines like this: "The story behind Nortel's fall."[2]

1. https://static.cambridge.org/binary/version/id/urn:cambridge...

2. https://www.theglobeandmail.com/report-on-business/the-story...

They might had delivered some network equipment of modest importance (in the period 2005-2010), I was referring to when they started becoming prime contractors.

> BT replacing Marconi with Huawei backbone routers for its 21CN project, that was early 00s.

Just looked this up: Huawei was selected by BT in 2005. That's already about 4 years after Nortel's initial collapse, and a year after the SEC began investigating Nortel for financial fraud. Again, this illustrates how blaming Huawei for Nortel's fall doesn't make chronological sense.

I worked for Nortel for 3 months. When I returned for work after Christmas, we were sent to the Holiday Inn and told we were retrenched. I ended up with a great payout AND received a $14K performance bonus on top a month later!

The short answer is no. It didn't help, but it wasn't the only thing.

1. Around that time, I worked for a telecom startup down the street from Nortel in Kanata. It went bankrupt after Worldcom dupped us into thinking it was buying our tech. It was a lie, and Worldcom was cooking the books. Telecom was starting to implode. Who knows who else was lying.

2. EVERYBODY in Ottawa knew someone that worked at Nortel. How could one, or a few hacks, take the work product of thousands of thousands of people and create another products. Heck, I can barely understand some of the open source products, and they have docs! Maybe they solved a problem of how some things were being done?! Either way, it's one thing to get ahead, it's another to stay ahead.

3. My good friend at the time was one of those that went down with the ship. Lost all his stocks and years and years of work at Nortel. He was a senior engineer and he told me mismanagement was starting to happen from the late 80s. Even in the late 90s, Nortel was throwing so many parties, and hiring anyone. They were just burning money. We always wondered where did they get all the money and how could they afford to hire so many people that barely qualified?!?

I don't claim to have all the answers, but like most things in life, it's not one cut that kills you, it's the other 999.

IP theft or not, the entire tech company was hit very hard around that time. Telecom was especially hard hit. Nortel was hardly unique. Disclosure: I'm in that business, work for Ciena which now owns one if the remainders of Nortel. Opinions are mine. Sycamore Networks: From $45 Billion to Zilch: https://online.wsj.com/article/SB100014241278873239261045782...

Telecom's big fall: https://www.baltimoresun.com/news/bs-xpm-2002-04-07-02040600...

An often overlooked element was how poorly Nortel treated their Chinese employees. State directed espionage doesn't get hundreds of engineers to jump ship to domestic PRC firms. Money does. Then the knowledge travels with them.

Nortel Guangzhou had bamboo ceiling on Chinese management roles in the late 90s that extended to even Chinese Canadians. Big surprise when entire engineering teams got poached by ZTE and Huawei. A Chinese Canadian aquaintence who started in Nortel Ottawa poached most of his Nortel GZ engineering team because state companies simply compensated better and provided more opportunities. A few years later, he poached a bunch of engineers from state company back to a foreign one who promoted. PRC talent would rather stay working with foreign multinations due to prestige and better conditions especially in the late 90s, but at the end of the day, money talks.

There's this quote from the article: “None of Huawei’s products or technologies have been developed through improper or nefarious means.”

And then there's this WSJ article: "Huawei Technologies Co. said a small portion of its router software apparently was copied from Cisco Systems Inc., but the Chinese company said it is removing the tainted software from its routers globally." https://www.wsj.com/articles/SB10485560675556000

Oh please. Everybody remembers when those two guys got arrested for stealing Samsung's folding display tech and selling it and a year later Huawei has a folding smartphone.

Having trouble finding sources but I was under the impression Samsung made the display for Huawei's phones.

Edit: Nevermind looks like it was made by BOE. https://pocketnow.com/the-new-huawei-mate-x2-will-not-use-sa...

"[Arrived] in a $1-million leather bodysuit with an anatomically-correct gold breastplate and a 15-carat-diamond nipple. 'You were just surrounded by the most interesting and intelligent people that you could find anywhere in the world,'"

Interesting juxtaposition haha.

IIRC you're referring to the wife of Michael Cowpland, who founded Corel. They were another hot tech firm at the time in Ottawa, but not related to Nortel.

I was going to quote this. Totally insane. Even with a gold breastplate and the diamond, I cannot fathom how this managed to cost 1 million.

I lost so much money when Nortel died. It was unimaginable at the time. It’s like Cisco going belly up overnight, it would be so shocking no one would believe it.

I would be surprised if Cisco went bankrupt overnight, but to me their future looks quite bleak. Every day less companies need to buy their hardware as more move to the cloud. The biggest tech companies can build their own hardware. And the hardware isn't extremely technically advanced enough to prevent commodity competitors.

So one day if the revenue begins shrinking and they double down by going into debt to expand, then yes they could be facing bankruptcy just like Nortel.

Cisco isn't the company you seem to think it is. They've spent the last ~15 years gradually diversifying away from that single point of business failure in their old line hardware business. They recognized the threat of cheaper competitive hardware products a long time ago, China gave them a serious wake up call (the issues with Huawei go all the way back 18 or so years now).

The majority of their sales come from software and services now. The business you are likely thinking of when you picture Cisco, has been shrinking. Growth in their other segments (whether organic or acquired) is keeping them flat more or less across the past five years (both sales and operating profit have been close to flat going back to fiscal 2017).

You know a company has become truly successful when nobody knows exactly what they do. E.g. SAP

IBM too

There’s so many niches in enterprise software just looking for someone to say yes to all of their questions. It’s not surprising really.

I was surprised when Cisco bought Tidal enterprise orchestration management tool (which I had worked with for years). Great way to get your foot in the door at a lot of Very Large financial companies offering professional software services. They've been buying up tons of B and C tier enterprise software for years, and a lot of blue chip clients in the process.

Listened to a podcast about them recently interesting start of a company (in a bad way). Talking about the founders that were kicked out.

This might be counter-intuitive but this market is still growing. Random reference from a quick Googling: "NEEDHAM, Mass., June 9, 2021 – The worldwide Ethernet switch market recorded $6.7 billion in revenue in the first quarter of 2021 (1Q21), an increase of 7.6% year over year. Meanwhile, the worldwide total enterprise and service provider (SP) router market revenues grew 14.4% year over year in 1Q21 to $3.4 billion. These growth rates are according to results published in the International Data Corporation" ...

"Cisco finished 1Q21 with a year-over-year increase of 3.4% in overall Ethernet switch revenues and market share of 49.3%. In the higher-speed segment of the Ethernet switch market (25Gb/50Gb,100Gb & 200Gb/400Gb), Cisco is the market leader with 40.9% of revenues. Cisco's combined service provider and enterprise router revenue grew 18.5% year over year, with enterprise router revenue decreasing 1.3% and SP revenues increasing 33.2% year over year. Cisco's combined SP and enterprise router market share stands at 37.6%."

A lot of cloud growth is reflected outside the cloud. Corporations still need networks even if they do some things in the cloud. There's generally just a lot more traffic all over the place.

This isn't really commodity. I work in this industry. a 400Gb switch isn't something you just build yourself or buy at your local electronics store. There is definitely competition but it's not exactly what I'd call commodification. What is true is that some of the building blocks are available...

There's definitely a range of future outcomes for this company but it's not gonna be the cloud that does it. Could be another bubble burst or major economic downturn combined with other factors (which I guess was sort of the Nortel story, though I doubt Nortel was sitting on as much cash as some of their contemporaries).

>Every day less companies need to buy their hardware as more move to the cloud.

That's not even Cisco biggest problem. Businesses still need to buy networking equipment, and there a more small datacenters than ever, but they aren't buying Cisco anymore. 10 years ago, Cisco was pretty much your number one choice, now they can't really compete. I'd be surprised if we have much Cisco equipment left in five years. It's almost like their R&D just stopped. Our next green field networking project will be completely without Cisco equipment, and we've been a Cisco partner for 15 years.

Yea, many Canadian mutual funds were heavily invested in Nortel. I remember family members saying they lost a good chunk of their investments when Nortel went belly up.

That's some serious home bias.

At its height it was 1/3 of the valuation of the Toronto Stock Exchange

How many American's invest in Apple?

That's Canada. Small market, but many invest 50% or more at home, which gives huge exposure to mostly oil&gas and banks.

No, the bursting of the tech bubble and financial scandals killed Nortel.

Huawei wasn't even a major player outside of China back when Nortel fell into trouble. From 2001-2004, Nortel's sales collapsed, it began downsizing, and the share price plummeted. In 2004, the SEC began investigating the company for financial fraud (misstating earnings).[1] Huawei only began to expand its overseas sales in 2003-2004.

The idea that hackers (whose identity isn't even known) brought down Nortel is just post facto rationalization of the mismanagement of the company. When the article says the hack began, in April 2004, Nortel's stock price was already down by over 90%, and the SEC was just announcing its investigation into financial irregularities at the company.

1. https://www.reuters.com/article/us-nortel-timeline-sb-idUKTR...

nope the earliest they could tell the hackers where in the system was 2000 or earlier not 2004... CSIS warned them many times there was espionage activity but it looks like Nortel didn't take it seriously. They even found listening devices in their building after the company was gone.. during this same time they even clone a cisco router down to the same typo's

having to compete with a company that has the backing of a state government and that has access to your plans and contract bidding pricing it wouldn't be hard to take them out.

Huawei was not a major international player throughout this era. Competition with Huawei is not what led to the collapse of Nortel. Nortel's mismanagement and financial misrepresentations are its own fault, and trying to blame Huawei - which was barely on the international stage at that point - makes no sense.

A giant usually doesn't fall because of outside forces because like they say barbarians are always at the gate. It usually mismanagement on top of mismanagement until they can't pay their bills anymore.

I think this article title says it all "DND may abandon $1B move to former Nortel site because of surveillance bugs"

https://www.ctvnews.ca/canada/dnd-may-abandon-1b-move-to-for...

I'm not seeing this 2000 breach, but even then by 2001 they had laid off 2/3rds of their workforce, and faced repeated internal scandals after that.

Nortel's campus was full of listening devices https://www.infosecurity-magazine.com/news/was-nortels-ottaw...

I’d take that one with a bit of a grain of salt - significant factions in the DND really didn’t want a hand-me-down facility in west Ottawa, they wanted to build something new across the river or downtown. There may have been strong inventive for some to “find” reasons to not occupy the former nortel campus.

The DND changed their story a few times, but the latest AFAIK is that they deny any listening devices had been found. https://ottawacitizen.com/news/national/defence-watch/the-my...

The sheer amount of bugs planted in buildings, especially at time of construction, has been an astonishing revelation to me. It’s funny to think that this is a full time job but many people have no idea it happens.

It’s worth mentioning that, Nortel was filled with Chinese engineers, graduating from Waterloo and other Canadian schools.

Just see what percentage of students in Canadian schools are from China. Feels like Beijing every time I visit a Canadian engineering school.

It seems Canada has a problem with real estate too linked to Chinese investment.

I wonder if it’s any different in US or elsewhere.

in 2000? Going to need a source on this.

Previous discussion..

https://news.ycombinator.com/item?id=23743869

Betteridge's law of headlines applies: No.

I mean, from the description in the article of how the hack occurred, and how management basically did nothing to respond, makes it pretty clear that if weren't a Chinese government hack, it would have been something else. Nortel management appeared very "asleep at the wheel" and it seems pretty apparent the company was suffering from a significant cultural rot.

yea its sad, government warned them about espionage activity early on and they did nothing... imagine after all this they never checked for listening devices? or for network intrusions?

It's an interesting history lesson, but the problem with Huawei today is this:

They (and the CCP) will fund 100% of the rollout for a telecom to use their 5G server equipment. For Western mfgs. to compete, either Huawei has to be banned, or equivalent pricing and funding has to be instantly available. Or no sale.

Nortel never had a chance to compete with that.

The US funds Boeing's overseas sales via the Export-Import Bank, but that plan has to be extended to domestic and foreign 5G sales.

https://en.wikipedia.org/wiki/Export%E2%80%93Import_Bank_of_...

Note that on mobile at least the page autoplays a loud video ad (1 of 2 for me before I bounced), which when you scroll down to get away from it simply pops out and follows you.

Did not read the article as a result.

I rarely open articles on my phone (due to not having an adblocker). Some websites are terrible like one of them tries to stop you from being able to back out or scrolls you down to those bad articles at the bottom of many sites.

I installed Firefox Mobile + uBlock origin on my Android phone a few days back (after reading this HN discussion https://news.ycombinator.com/item?id=28631005) and it's amazing. Not a single ad or video in this page.

Did an American hack kill a Netherlands tech company? [1]

[1] https://en.wikipedia.org/wiki/DigiNotar

...or a 21 year old Iranian student...or the Iranian government? [1] The source you've provided isn't exactly clear on that point.

[1] https://en.wikipedia.org/wiki/DigiNotar

Interesting question: Should the government act as a red team and try to pen test the systems of organizations in its own country?

Recently read an article https://foreignpolicy.com/2021/09/24/china-great-power-unite... which presents the argument that China, a declining power that is desperate to regain economic momentum by "dominating critical technologies—such as artificial intelligence, quantum computing, and 5G telecommunications", and creating exclusive spheres of influence abroad, will likely start a war much like Germany in 1914 or Japan in 1941. Got me thinking that if the war was guaranteed, would it have to be in Taiwan or south east Asia islands. However, both areas are under a magnifying glass and highly contested. And sea battles do not favor China. But what if 声东击西

- Iran attacks Iraq, and China/Russia gives support? Would Nato/US still have the stomach to go back into Middle East? This would allow China to fully utilize its full ground army force and massive train network that now stretches into Central Asia and Europe. And if Iraq falls, Saudi Arabia, United emirates should fall easily, and those are oil countries that will give China massive resources and wealth. US is fairly self sufficient in terms of oil resources, but not Europe! Europe, under gas and oil resource stranglement, will acquiesce to any demands from Russia + China.

- Turkey attacks Greece on grounds of islands contestations. Again, this would allow China to fully utilize its full ground army force and massive train network that now stretches into Central Asia and Europe. After Greece, Russia can easily retake most of its former colonies, and give China a way into Central Europe.

> - Iran attacks Iraq,

bro, it's not 1989 anymore

1989 was other way round.

1979 was the other way around. 1989 would surprise me a lot.

it is meant to refer to the fact that they are no longer rivals; iraq is effectively an iranian vassal

sounds like western elites have retreated into their forbidden cities and only read false world reports. In what world is China a declining power if one thing its the US that is being isolated by its partners because the deal given is pure shit for the partners economy.

Like Kissinger said "it may be dangerous to be America's enemy, but to be America's friend is fatal."

Reply on news.ycombinator.com