Hacker News Reader

Comment by 11thEarlOfMar | Hacker News Reader

It's surprising to many, but these tools run primarily on Windows, and, without anti-virus protection because it can interfere with critical timing actions that the tools need to make. The IT systems are locked down like a fortress, but tool technicians can still bring viruses in and transmit when they connect their laptops for diagnostics and maintenance. I can easily see a virus running amok.

View on HN

Seem likely. The monitor shown at 1:09, 1:58, and 7:25[1] on the "TSMC Fab Tour" video on this page[2] looks suspiciously like Windows 95, but could be Windows Embedded I guess.

[1] https://imgur.com/a/J4EhzPC

[2] http://www.tsmc.com/english/newsEvents/dc_video.htm#

Looks like XP Embedded.

On a factory floor? It could be win 3.1. Because something is high tech doesnt meant the machine building it hasnt been repurposed a dozen times. Dos is still out there.

In our factory, is at least still one OS/2 PC running. I'm not proud for this.

last time I was in a 300mm fab I definitely saw OS/2, command lines being used by technicians, and a LOT of windows embedded 3.1.

The main defense is a computer at the entrance where technicians are supposed to scan their USB drives. It's a devils choice really...do you put the tools on an externally accessible network? If not how do engineers get data off of tools? how does the factory talk to itself and schedule things?

I don't envy FAB IT. Utterly thankless.

It's not necessarily something to not be proud of, though. NASA's space shuttles ran on something like a 386 CPU. That's what was available when the system was designed. Probably one of the best "if it ain't broke, don't fix it" examples. Swapping out to newer components would have meant replacing so much of the system, that it wasn't worth it. Especially if it was doing everything that needed to be done.

Same thing with your OS/2 system. As long as it's working, there's really no need to change it. The fun begins when it no longer works and needs to be replaced. How much of the factory will then need to be replaced/updated just because the controlling computer can no longer operate the older equipment? That's the type of scenarios that give managers nightmares.

Well, I think the problem arises out of the fact that most systems that size are "broke", and some other systems (a small QNX or OpenBSD [or maybe these days, some form of L4]) might have fewer flaws that can cause issues, the likes from which TSMC has apparently suffered.

OS/2 was very stable, and certainly immune to viruses designed for Windows. The only problem back in the day could arise from its infamous single input queue which could freeze the entire system if a single program didn't respond in time to GUI events. The problem however was corrected in the latter versions.

We've seen this before many times. It's an inherent weakness of proprietary software solutions: you're at the mercy of the vendor as far as porting it to a new platform. The weakness becomes more urgent as the platform grows old/unsupported/insecure/etc. It may have been Windows this time, but it could have been DEC or Solaris or what have you.

Open Source Software may or may not be easy or economically realistic to port, but at least the users have a say.

Software doesn't exist in a vacuum, it needs support and maintenance, which somebody has to pay for.

Can we stop repeating this as a strength of OSS? It definitely is not. "Users" have a problem they want solved, they don't really want that problem and to learn how to patch software.

If the "user" is the world's largest independent foundry, I'm sure they could find someone to patch the software they're using.

With OSS, it is at least possible for a new vendor to step in with competing support and maintenance if the old vendor is failing. It may also be possible for customers to participate in supporting and maintaining the software to varying degrees, for example by hiring the core maintainers; whether that is realistic depends on the size and complexity of the software, the tech culture of the customer, etc.

This isn't caused by lack of open source software, this is caused by lack of management, partitioning, and containment. I assume the issue was caused by a 100% flat network, with office worker's laptops being on the same subnet as million dollar fab machine computers.

Certainly those are also factors, but the central security principle of defense in depth argues for minimizing all possible factors. Having nodes that are highly vulnerable to viruses is a weakness and a contributing factor towards a breach.

Even an airgapped system can be breached, as we saw with Stuxnet. There's no perfect solution for protecting vulnerable systems.

There is benefit in making a negative outcome less probable, like there's benefit in wearing a bullet-proof vest though you can still be shot in the face.

Remove all data ports?

Upload instructions via punch cards

"Hey guys I found these punch cards in the parking lot"

You are kidding right?

The machines at TSMC would be at 100% utilization on million dollar jobs, management would never accept downtime for maintenance for software updates that could introduce bugs.

Its one thing for consumer devices to receive updates, its a completely different ballgame with industrial. Downtime due to anything can be millions of dollars per hour in losses. So it comes down to what could cause more monetary loss, downtime due to hacks or downtime due to bugs introduced.

"If it works why break it".

Its the same reason why there are production lines where physically, hardware could be replaced and improved to reduce wear, reliability, power usage, etc (even things like motors) but they arent, for decades. Only when things start costing money outside the original design do engineers perform upgrades.

Exactly. You can say "oh, just take the machine out of production for a bit do software patching". However that reduces the uptime for your tool which will require buying more of them to get the same throughput. More tools is not only capex, but also space on the factory floor.

This is extra true for my company. We sell sorters, which don't actually do anything useful in terms of processing the wafers. The tools just shuffle the wafers between FOUPs. While these tools help customers optimize processes by changing batch sizes and such, they really want to have as few non-process tools as possible. So uptime is really important.

> these tools run primarily on Windows, and, without anti-virus protection

Back when I made rocket things, we used an RTOS. Are real-time operating systems (like QNX) not widely used?

They should be (and are in some industries, apparently not in this one).

But QnX real-time capabilities are just one nice part of the mix, the user land drivers and impossible to crash kernel are equal partners in giving it the stability that it has. Process isolation for drivers should be mandatory.

Reply on news.ycombinator.com