Hacker News Reader

Comment by taspeotis | Hacker News Reader

I would be most grateful if someone could please explain what sort of tools are likely to be used here, and why a power loss to those tools would ruin days/weeks/months worth of output relative to the time they were offline?

View on HN

Semiconductor manufacturing involves a lot of precisely controlled processes. You put the wafers into furnaces and pass a gas over them for X time and Y flow rate at Z temperature to impregnate the wafer with various chemicals. You put them in low pressure plasma environments to etch them, again for X time at Y flow rate. There are half a dozen more of these as well, like applying metal and implanting ions.

These values are experimentally tightened to get the highest possible accuracy to the desired effect and improve the number of working chips that leave the factory. If the power cuts out you don't know what conditions the wafer experienced while the system was winding down completely uncontrolled and your processes haven't been designed for the wafer going through the ramp up twice.

The reason why it's lost so much output is because modern semiconductor processes have hundreds of steps and (I believe) a lead time in the months, so the amount of material that's in flight at any one instant has to be huge to get any reasonable throughput.

So they realized late that an early step of the pipeline was off, rendering everything that has been through flawed ?

More like many steps in parallel would all be affected by a power outage, and much or all of the wafers in progress were ruined or simply not economical to recover.

No, they experienced a power outage which has ruined every chip that was currently at any point in the pipeline. The realization would be immediately (as soon as the power goes out).

It said that it was 1/2 their production output for one quarter. Is that really possible for an instantaneous power outage?

If at that instant they have half there quarter’s product in various stages of processing (and there are hundreds of such stages, in hundreds of parallel pipelines) when all of those stages shit the bed, obviously yes.

That's the point. The only way that's possible is if things sit in a process that takes many months.

If you understand that that’s the way the process works, then why are you asking if it’s possible

or that the restart procedure after a power failure is going to take a few weeks...

I can imagine that if the factory is entirely automated and a full-restart has never been attempted. Every single machine will probably be in some bad state with unknown chemicals settled into unknown pipes in the machine, requiring custom flush processes to be designed, and in some cases machines might have to be replaced, which in a human-free clean room isn't easy...

Yes, that's exactly what's going on here. The processes last quite a long time.

Similarly, a drought that doesn't last that long (relative to the life of a big tree) can nevertheless kill that tree even though said tree has been growing for centuries.

In other words, there need be no correlation between how little time it takes to ruin something that takes a very long time to make.

Interesting. So what do they do with the materials afterwards? Landfill or melting or?

Probably varies depending on stage. Wafers are very pure silicon, so the ones that haven't yet been contaminated with trace elements would probably be melted down for reuse. Wafers later in the process are probably carrying a lot of material (gold/silver/platinum, maybe iridium, some heavy metals?) that would be worth salvaging in bulk.

Acids, the manufacturing basically is controlling the etching process to get rid of unwanted parts and keep the designed metal circuits.

When you lost power, you are not sure if the chips stayed with acids for too long or too short, or coated with unwanted amount of materials, the uncertainty kills the yield rate, which can be already low since memory chips require repeated stacking nowadays.

Yesterday this issue was being discussed in /r/DataHoarder and I asked a similar question:

https://www.reddit.com/r/DataHoarder/comments/c6mt9l/a_13_mi...

I imagine that the process is very highly pipelined and optimized, and I would imagine that they had some sort of backup (generator) that failed.

One analogy is to think about if you had a batch script that you were working on that touches a lot of files (1000s). Now imagine power was cut and the batch script was interupted because the computer turned off, but that computer hasnt been turned off in a long time (say it was a server).

First, you have to turn the server back on after a power outage. Was there corrupted files in it? You have to now get that server in a known working state, and if you have kept it on for years....then you may be in a world of hurt.

NOw you got your server up and running. You have the option of going through each of the 1000s of files your script was working on....but that will take time. Does it make sense to start from scratch? You will have to through our all the files you were working on, but at least you can start that script again. You could attempt to salvage every file, but that will also take time too.

Another analogy is that you have a bakery that produces soufflés on an industrial scale.

These soufflés take two months to bake. The baking has to be done in such a precise fashion that even fractions of a degree in variance results in the entire batch being ruined.

Worse still, it takes a long time to bring the oven up to temperature and stabilise it on the precise temperature. You can't just scrap the batch that's ruined and start production again.

I like this analogy. Lets expand.

These soufflés take two months to bake but you need soufflés every day for sales. What does this mean?

You always have 2 months of soufflés in various states of production at all times.

Now you lose power and all these as very fragile soufflés in production are lost because of the power failure. Furthermore, it will take you two months to get the first soufflés off the restarted production line.

Reply on news.ycombinator.com