Power Outage at Samsung Fab Destroys 3.5% of Global NAND Flash Output for March | Hacker News Reader

Power Outage at Samsung Fab Destroys 3.5% of Global NAND Flash Output for March

anandtech.com

150 pointsjonbaer104 comments

This sort of thing shows how scarily centralized electronics production is. If a disruption at one plant can take out 3.5% of global production, how many of these plants are there? On the order of 30? Imagine a terrorist group decides to bomb them all simultaneously; there goes the production of every device that uses these chips, potentially for years. Are they needed for something critical? How many products do we rely on that all come from the same dozen or so places?

I remember when Thailand had heavy floods which spiked the cost of hard drives for years.

Here's the first article I found about that incident, claiming HD prices shot up 50% in one week.

http://www.tomshardware.com/news/seagate-wd-hard-disk-drive-...

A large portion of that increase after about the first 3 months was driven by the manufacturers, not actual capacity constraints. They had an excuse to raise prices so they did. I wouldn't be surprised if there were a lawsuit at some point similar to the monitor and dram collusion lawsuits.

Not sure what you mean "manufacturers, not actual capacity constraints". The beauty of price signals is that they distill all the market information down to a simple metric. It incorporates increased demand, decreased supply, availability of substitute products, predictions about the future and more.

A spike in prices is the best way to ensure that the supply is increased ASAP. The spike is a strong indication that the market wants more and it makes it more likely that suppliers will make decisions to satisfy that demand.

Higher prices allow companies to justify overtime, urgent delivery of supplies, rapid decision making on contracts (i.e. not shopping around) and so on.

Or it was due to collusion in a niche controlled by a duopoly (WD and Seagate).

Also captured in the price signal. The higher price (in your example created via price collusion) is a signal to competitors to enter the market.

Not really, because the price increase sint actually due to real costs on Seagate/wd's side - everyone knows they will just drop prices if a competitor joins that's not part of the collusion. You'd be entering a market with huge startup costs, and once you'd ramped up production prices would drop and your margins would be razor thin.

The hard drive market is widely believed to die with the advent in solid state disks.

With a huge capital cost to get into the market, no investor will put money into building new hard drive factories when it will probably be obsolete in 5 years.

Because you can just hop right into the HDD market? Are we pretending it wouldn't take tens of billions of dollars, 5+ years, and patents nobody is willing to sell or license?

I don’t think the terrorist threat is likely. It would have to be on the scale of 30 simultaneous 9/11 attacks. Anything less would just be a temporary inconvenience, 3.5% for a month is only 0.3% of annual production.

Also, a group capable of 30 simultaneous infrastructure attacks anywhere in the world with the goal of disrupting civilization can easily find far better targets than chip fabs, and various intelligence agencies will have infiltrated such a large international group anyway. The notion is interesting but unrealistic.

Terrorists are a simple-to-state example of the problem, but if that doesn't sound convincing, remember that most (or all?) of these fabs are in a region dominated by a superpower that is moving towards dictatorial authority [1], and additionally, that this region is also (one of the many) moving towards a pattern of extreme weather events [2]. There are many possible events that could seriously disrupt such a concentrated industrial base, and only one would have to happen to cause problems.

[1] https://www.theguardian.com/world/2018/mar/17/china-reappoin... [2] https://phys.org/news/2017-02-china-severe-weather-patterns-...

I think a more realistic (and worrisome) scenario is war brekaing out in South East Asia. Fabs should be clear economic targets for opposing forces. There are a number of conflicts which would likely lead to severe disruption on global semiconductor market.

What thirty targets with similar security as chip fabs do you have in mind? I have a hard time coming up with anything other than nuclear plants for massive damage, but they have somewhat stricter access control, I hope. Oil pipelines are much quicker to repair and most countries have strategic reserves. Refineries maybe?

Liquified natural gas depots. Igniting one of these or a full LNG ship in the river or bay near population centers would kill tens of thousands and the fire could not be extinguished, we would have to let it run its course. Nuclear plants have a bit more security than a chip plant and the logistics of damage by nuclear plant takeover kind of require the authorities to do nothing for a long time.

What about critical infrastructure? Dams which could flood big areas are one example which would be absolutely destructive

I don't think flooding one area is on the same level as killing a large part of microchip production for a couple of years. Almost everything has a chip inside these days.

> various intelligence agencies will have infiltrated such a large international group anyway

But will those organizations get in each other’s way and thus cause us to not stop it in advance. Watch or read The Looming Tower for more context. I’m not sure we really learned our lesson for that one.

well, except 9/11 happened so things can happen that the world's intelligence agencies don't know about. China or the us or russia or a few other countries could just send out missiles in 5 minutes and accomplish this. so just assume idiots or madmen don't get in charge of countries. And it already happened.

Superpowers with huge Arsenal’s could do massive damage. I don’t think this is news to anyone though.

If the Korean War restarts (which is less likely than people think, but still more likely than simultaneous terrorist attacks), the world is likely to lose most South Korean export products for a period.

NAND chips are needed in basically everything electronic now. Phones, tv's, watches, ssds, microwaves, cars, medical devices, and a million other products. They should have become a low margin commodity a long time ago. There is price fixing afoot but the SK president deposed for taking bribes cancelled the investigation into it, and SK let the Samsung CEO out of jail after being convicted of bribing the government so it's going to get worse before it gets better.

The cost of RAM is already very very high. This has been surging for quite sometime and now we might see the same for SSDs too. Not to call this news as false, but its sad to see the price increasing :(

https://www.reddit.com/r/buildapc/comments/73el5s/are_ram_pr...

We've already been in it for SSDs for a while. We're just getting back to $40/120GB SSD Sata drives. And that's mostly via on-sale prices.

(diclaimer - this is mostly via newegg/amazon pricing - hard drive prices are something I follow in the background, this isn't a scientific/well researched observation by any means, but other folks I've talked to have similar observations)

I recall paying around $1/gb in ~2006, and that was for a 5400rpm drive (and I’m sure the older members here will say something in the mb range). So while i agree, the rate we have come to expect value to increase isn’t continuing on that exponential curve we experienced the last 20 years, the prices are not expensive by any means.

If you’re talking mechanical drive prices, the best deal I’ve seen around are the wd easystores sold at Best Buy. When they go on sale for under 150 for the 8tb version, hard to beat that value. I filled my entire storinator up with them.

Can’t wait until ssds can compete with price/capacity.

2006 isn’t relevant to today’s market though. We can’t go back in time. Today’s businesses are built on data set sizes, compute loads and software footprints that in many couldn’t run on 2006 scale hardware at all. Unavailability of modern scale systems wouldn’t set back many businesses, it would kill them.

The biggest risk of supply shocks like this, even modest ones, is that businesses will decide to buy up supply early to avoid any problems getting it later. That further restricts supply for everyone else, driving prices even higher, and soon you have a ‘bank run’ on that resource choking off supply and making it unaffordable. This happened in the early 90s to RAM. A fire at a plastics factory for material used in RAM chip production created a run on RAM, even though in reality it wasn’t a threat to supply. The fear of a shortage created a shortage.

As a non-observer, what timescales are we talking about here? These statements don't mean much without context. The last storage price spike I recall was the great 2011 Thailand flood. Have there been fluctuations the common man should know about since then?

Roughly: a flash memory shortage started in late 2016 when everybody but Samsung was having trouble transitioning to 3D NAND flash memory, and Samsung was having trouble with their third generation of 3D NAND. All of the flash memory manufacturers are now producing 64-layer or higher 3D NAND and prices have been coming down for a few months, but they're still higher than 2015 prices.

I think stabilize is a better word, I hardly call a few percentage changes "coming down"

Consumer SSD prices are absolutely trending down. Everyday Amazon and Newegg prices are slightly better than the Black Friday/holiday sales of four months ago.

Strange the NAND price didn't drop that much. Weak demands of SSD?

*NAND price and SSD price never quite trend together or match up.

i paid $75 for a decent 240gb SSD back in 2014.

The price for a similarly decent 240gb SSD is still around $75 today.

Back then I thought 512gb/480gb would be available for $75 in 2017.

Capacity is the same, but how about durability of performance of those drives?

Worse nowadays. Every new process shrinks nand cells, on top of that we are on TLC everything now, while 5 years ago huge SLC chips were norm.

Good 2012 drive was Samsung 830 surviving 10 Petabytes, nowadays consumer drives are expected to survive ~100TB.

It wasn't that long ago that ram cost closer to $100 a gigabyte.

Yes and about half way between then and now it was about $100 for 16GB.

I remember paying $450 wholesale price for 8mb of ram. That was 25 years ago, which doesn’t seem that long ago, but in tech that’s centuries ago, so I’m not sure what point you were trying to make.

That $10 per gigabyte isn't "very very high".

Even if it does represent a doubling from the bottom.

Power can be critical in many ways: loose it at a meat processing plant and all production is lost too.

Reminds me of another situation where power loss was... a huge problem: http://abcnews.go.com/US/texas-chemical-plant-braces-explosi...

The Fukushima meltdown was ultimately due to power loss as well. The tsunami started the ball rolling, but the simultaneous loss of power from the reactors, the backup generators, and the grid is what led to disaster.

USCSB video explaining the causes of that explosion: https://www.youtube.com/watch?v=WZmUVQMh9qM

"Power outages tend to happen on various semiconductor plants and at times they harm production wafers."

One of the most meaningless sentences I've read in recent memory. The journalist didn't bother to ask how often they have power outages and how often those harm wafers, but they still put this in the final article?

You don't see any value in pointing out to the lay reader that such events aren't unheard of, unless their frequency and severity is precisely quantified?

It's not very hard to come to the conclusion that this might be done deliberately...

It's hard for me. Help me out.

There have been many instances of price fixing and collusion among DRAM, Flash, and storage manufacturers going back to 2000 and before.

If however you can attribute the price hikes to a natural disaster, say a flood or power outage, then you'll receive less scrutiny for sustained price increases.

To whose benefit? You mean sabotage by a competitor?

Samsung's, if they have a tight enough grip on the supply. It strikes me as somewhat unlikely because I thought the semiconductor industry was more competitive than that, but I don't know enough to rightly judge.

It brings to mind those recordings of Enron traders literally giggling as they ordered a power plant offline to drive up prices during the infamous California rolling blackouts. Yee-haw, value creation!

Meh, they could just produce the stuff and not sell it, if they want to reduce supply.

If they make it, and not sell it, now they have to store it, insure it, and secure it. And then if they have too much, and run out of storage space, they now have staff with nothing to do.

If they're allowed to by contracts and the law, sure.

Is there a law saying you have to sell a product? Seriously asking, I've never heard of that and it sounds odd to me but honestly I have no idea.

Contracts can absolutely require you to deliver something in the future. That's often the entire point. Penalties for failure to deliver often depend on circumstance -- e.g. "our bargaining position got better and we want a higher price" isn't an acceptable excuse but "oops, looks like an act of god destroyed your chips, now you'll have to pay the higher price" is.

Laws play a role too. The most egregious market manipulation tactics are often illegal, especially when monopolies are involved. Of course, this doesn't do much to dissuade the manipulation itself, but it does make companies jump through hoops to disguise what they're doing. Samsung can't send a video of someone in a Darth Vader costume saying "The deal has changed." to their customers, because that would be a smoking gun and easy to prosecute. "Oops, 3% of the product was destroyed, looks like prices will just have to go up" achieves the same end but keeps plausible deniability in tact.

Or it could be a legitimate accident. There is certainly no shortage of things to go wrong.

Just remember: Hanlon's razor is trivial to exploit. Live by Hanlon's razor, die by Hanlon's razor.

There are some things which manipulating supplies are not allowed. These typically are foodstuffs, water, medcines, and other basic needs, under some circumstances.

I'm not sure Electronic components fall in that category.

Then that needs to change. Flash storage is an essential commodity.

Baloney.

There are great many people who would die if the price of their food, or their water, or their medicine doubled tomorrow.

If the price of flash doubled, many people would have to choose between less storage, slow storage, or less money for other things. But nobody will die.

In fact, I bet the intersection between the people whose lives would be at risk from a sharp increase in food, water or medicine, and the people buy any amount of flash every quarter is nearly zero.

Imagine that you run a charity. Your job is to save lives. Should you spend every penny you get on medicine and spend nothing on IT? Of course not. It's impossible to run a charity like that. You must have computers, and you would prefer it if they are fast and reliable. You choose to divert some money from your programs to have efficient administration.

Now imagine the cost of flash doubles. You're forced to choose between "less storage, slow storage, or less money for other things." All of those options mean more death.

Flash storage can be more important than medicine for the sick or food for the hungry.

thats much harder to justify to regulators and inquests.

Makes a great case for on site energy storage.

They have it, but not enough, probably because they would need a lot more. From the comments of the article:

"Korean media has coverage of this.

Battery powered emergency power did kick in. However, some areas had to be shut down because of the amount of power that battery power can cover. Stuff that cannot be shut down were kept operational.

Not all the wafers will be thrown away. They are checking which are safe to use. Maximum damage is estimated at about $50 million."

Maximum damage is estimated at $50 million = 3.5% of global supply for a month

global supply for a month = 30 * $50 million = $1.5B

global NAND market = $18B/year

That seems low to me. Are there other figures on the value of the global NAND industry to compare this with?

$50B is likely their wholesale value, not end user value.

> Maximum damage is estimated at about $50 million.

If that damages figure is correct, I'm amazed they didn't have backup generators.

More like a case for on site power generation. Semiconductor fabs consume enormous amounts of power. Would be difficult to handle with energy storage systems, as least with today's technology.

I was insinuating Tesla Powerpacks, similar to the Hornsdale Power Reserve.

EDIT: After doing more digging and seeing how much power the facility requires, energy storage is definitely not an option. They'd need their own natural gas turbine on site and redundant transformers (a transformer failed).

Right, but even those might not be sufficiently energy dense because industrial processes use so much power.

Diesel generators could easily handle their load requirements.

There's a row of new(ish) datacenters in Santa Clara (Silicon Valley) where there used to be a row of fabs -- because a large amount of power infrastructure was already available after the fabs were torn town.

All of these datacenters use energy storage to tide themselves over power failures.

Why is it that energe storage works for datacenters but not for fabs?

My guess would be fabs take a lot more energy at peak draw- they have to melt the components in a chip, and data centers are trying not to melt them.

Sure, but, do they need to keep producing at peak when there is a power outage?

You must have a plan to ramp down to a steady state of consumption that keeps things safe. Even if they need many megawatts, the 50 million in damages figure quoted elsewhere buys a hell of a lot of generators.

My layperson's understanding of the furnaces producing silicon ingots is that they can't power down in-process; best case, they take weeks to bring back into service, while the worst case is serious damage to the equipment as the silicon inside recrystallizes.

I wonder if it's even possible to ramp down this single process without a similar hit to productivity as an uncontrolled power loss, never mind the other stages in the fab process. If not, fabs can consume tens of megawatts, so what does it cost to build and maintain the necessary backup generation capacity to weather a 30-minute outage?

One interesting example: USD $1.5 million for 10MW of flywheel-based backup capacity (http://ir.p10industries.com/news-releases/news-release-detai...) - but these will last ~15 seconds under full load.

> so what does it cost to build and maintain the necessary backup generation capacity to weather a 30-minute outage?

Tesla's install at the Hornsdale Power Reserve [1] is 129 MW of storage capable of discharging at 100MW. This could supply power for far more than 30 minutes if an outage occurs. Estimated cost is $50 million, which appears to be roughly the same cost as this manufacturing failure, so there's a middle ground to be found.

[1] https://en.wikipedia.org/wiki/Hornsdale_Wind_Farm#Hornsdale_...

> but these will last ~15 seconds under full load.

Flywheels are an alternative for battery backups. They are used to give enough time for generators to come online and stabilize.

As a layman, can you explain why a fab is producing silicon ingots? Instead of consuming them. Because they are two different things.

Nope - silly assumption on my part.

Do you have any evidence that a fab, which processes many wafers per day, really has significant peak/low draw differences?

By energy storage, I assume you mean the energy stored in diesel, and released when combusted and used to turn a generator. If you are referring to batteries, do you have any links? I would be really interested to learn more about large scale datacenters that are using something other than diesel/natural gas for backup power.

I'm mystified why you think energy storage is only diesel. Any datacenter-focused energy storage resource you'd find on the web doesn't mention diesel for short-term storage. Batteries and flywheels are the most popular. Diesel is a medium-term solution.

I've never bought capacity at a datacenter which only had diesel backup. It simply doesn't start fast enough for short-term capacity.

When I read the comment about energy storage, I was thinking for longer term backup power of greater than 60 seconds of runtime.

You are correct, that it takes some time, usually about 10 seconds for a diesel generator to come online and provide the power. That power gap is traditionally provided by a UPS (which can be chemical (battery), kinetic (flywheel), or some other short term power supply).

Again, my interest is diesel alternatives that are being developed and deployed in large scale datacenters.

I used to work at a datacenter that had all the usual elements. ATS, UPS, Caterpillar genset. It's all quite interesting.

Ah. Well, one up-and-coming alternative is Bloom Energy fuel cells, which run on natural gas, and are supposedly inexpensive enough producing energy that it's reasonable to use them as "peaker" power production. I've never seen any numbers for them, though.

Microsoft: http://www.datacenterknowledge.com/archives/2012/09/17/micro...

I mean, it's sort of silly to locate your data centers where you can't use on site renewable generation, no? If you're going to put a datacenter somewhere due to low land costs, but still meets your user latency requirements (speed of light), if you can generate onsite with solar and meet your consumption needs, and battery back it, and have utility power, you don't need large diesel gensets for battery backup (gensets that are rarely used but depreciating constantly).

Apple built a 200MW solar farm to power its Reno, NV datacenter. It also built a 50MW solar power plant on 300 acres in Florence, Ariz., which is used to power its Mesa data center. They use 100% renewables to power all of their data centers.

https://www.computerworld.com/article/3161732/sustainable-it...

And in manufacturing, Tesla is covering the entire roof of Gigafactory 1 with solar. It can be done.

Is hundreds of acres of solar an option in South Korea?

It appears so.

https://www.koreaexpose.com/solar-agriculture-s-koreas-new-f...

I’m gonna go out on a limb here and guess that you’re not an electrical engineer.

I don't need an EE degree in order to calculate peak and continuous energy use for a load.

Reminds me of that epoxy plant disaster (can't remember/find source atm) that shot prices through the roof.

Is it this Japanese chemical plant explosion from 1993 perhaps? http://www.apnewsarchive.com/1993/Semiconductor-Industry-Fac...

I think that's it!

Or Enron.

These guys should think about a Tesla battery UPS for their risk mitigation plan.

Formerly discussed here https://news.ycombinator.com/item?id=16594694

Most likely silicon cruicibles. The reason there is no backup power is because this would've taken building an own power plant.

Ok people keep saying there was no backup power but there was, it was just not enough to power everything. Source: http://m.yna.co.kr/kr/contents/?cid=AKR20180316130000003 maximum estimated amage was around $50mil usd. It says it's their internal transformer that failed. Up to 60k wafers were affected but not all of them are unusuable.

Not likely. I don't believe Samsung makes their own bare wafers and I know of no fab that makes wafers on site. I don't think the demand model would work to make that vertically integrated. A couple crucibles can easily supply a Fab and the bulk cycles of crucibles is uite different from the more unitized fab production.

Well, this is the only explanation I have for loss on such scale, and the fact that wafer manufacturing is the only thing that can't be usually backed up with UPSes.

All DRAM fabs use a lot of proprietary pixie dust treatment for wafers these days, and it is not looking improbable that they had a wafer manufacturing on site.

If they were to loose power on a stepper line, they would've only lost wafers being currently processed.

On my memory, there was a number of wafer shortages when Taiwanese wafer makers had similar incidents. As far as I was told, if a crystal forms on a crucible wall, touches it, or moreover the whole thing freezes, a big part of equipment becomes a write off, and after that it will take few months for a company to resume production.

Lots of big power users have their own power plants, with the grid as backup. This is especially good if there's someone nearby who can use the waste heat from the power plant. Looks like the melting point of pure silicon is high enough that it's way above the temperature of the power plant's waste heat.

Link? I've not heard of those before, and am interested to learn more!

When I worked at what was then Inland Steel (Indiana, US) they had a coal fired 60 Hz power station. It did not produce all of their power but it did produce a significant amount. The reason it was there was that they could use steam after it had passed through a portion of the turbine for other plant processes.

Chip fabs are in a different situation. My understanding (and I am far from an expert) is that typical power interruptions such as the momentary loss when a breaker trips and recloses will interrupt production. A quarter second might as well be 30 minutes as it will cause all motor controllers to drop out. Chip fabs need something akin to a giant UPS (or alternative power feed that they can switch to in a few milliseconds. Such equipment exists and is costly. The business case is the cost of the power conditioning vs. the expected lost production due to potential outages.

Perhaps Samsung did have something that could ride out a short outage but not the one they faced. In that case there is a vast difference between 1/4 second drop and a 30 minute outage.

They're just the things that the silicon crystals are grown in. The furnace cools off and the whole thing needs to be re-started/replaced.

It does not work like that. The whole thing needs to be a single crystal. If a minute temperature change creates a second grain in the melt, it is all gone, along with the crucible.

Is there any place online, on a book, you suggest I go to in order to watch or read more about this? That all sounds remarkably interesting, especially for something which impacts all of our lives so much.

https://youtu.be/8QKzS_w_Ko0

Seems like this is going to develop into another years long excuse about why NAND prices remain high. Totally not collusion though, absolutely not.

Reply on news.ycombinator.com