Hacker News Reader

Comment by jmmv | Hacker News Reader

Original author here.

Back in September, I got the recommendation to read the "Showstopper!: The Breakneck Race to Create Windows NT and the Next Generation at Microsoft" book from some other comment thread here in HN. I don't exactly remember which thread that was though... but learning about the history of Windows NT sounded interesting, so I bought the book right away and have been slowly reading through it over the last three months.

The story in the book seemed so exciting (and messed up) that I ended up writing a "book review" with commentary on various topics that caught my attention. The details might not be super-precise, but they are hopefully enough to paint the right picture in its context. I thought you might enjoy it due to where I got the suggestion from!

View on HN

I have just been dealing with problems of VMS this week, and I have Dave Cutler to thank.

We have a temporary emulated VAX running VMS 7 and Rdb [a SQL database now owned by Oracle]. After several tries to move the Rdb backup files through our QA firewall with FTP, we finally had to use Multinet SFTP2 to push the backup instead (upping the transfer time from 3 to 40 minutes), which autodetected "VMS mode."

There are some secret record blocking/delimiter capabilities in the ODS-2 filesystem used by VMS that an FTP binary transfer will not preserve, and which the Rdb backup/restore process absolutely needs.

Thank heavens that UNIX thinks a file is a single stream of bytes.

Windows continues the feature experimentation with NTFS "alternate data streams." My, how clever that is [not].

EDIT: from the article, the claim "The open-source BSDs and Linux had just seen the light of day." is not quite correct. Bill Joy was involved in the earliest BSD repacking of v6 Research UNIX, shipped in 1978.

"In 1975, Ken Thompson took a sabbatical from Bell Labs and came to Berkeley as a visiting professor. He helped to install Version 6 Unix and started working on a Pascal implementation for the system. Graduate students Chuck Haley and Bill Joy improved Thompson's Pascal and implemented an improved text editor, ex. Other universities became interested in the software at Berkeley, and so in 1977 Joy started compiling the first Berkeley Software Distribution (1BSD), which was released on March 9, 1978."

https://en.wikipedia.org/wiki/Berkeley_Software_Distribution

...I have also read elsewhere that a conscious decision was made in Windows to never, ever support big-endian architectures (but I am not certain that this is correct). NetBSD and Linux can run big-endian.

"Now, I know you will complain that NTFS is slow… but you know what? It almost-certainly is not... NTFS on its own is fine..."

XFS appears to trounce NTFS, and it consistently leads the performance scores on tpc.org; until very recently, SQL Server on XFS/Linux was well ahead of Windows.

You're blaming tools that don't support the full featureset of one of your platforms.

VMS's ODS2 filesystem evidently has concept of files that are grouped in records. The ftp program that you use on VMS wasn't designed to support sending that info to a destination filesystem. Probably because the original Unix filesystem only supported the idea of one stream of bytes for a file. You'd need an FTP server on the destination side to support records seamlessly if you wanted full fidelity.

That's why NTFS supports alternate data streams - the MacOS HFS filesystem supported data and resource forks for a file. An NTFS alternate datastream was where the resource fork was placed so the NTFS user would only see one file from the CLI and the GUI and the builtin copy tools would copy all streams. deleting such a file would remove both streams on NTFS.

Ok, now I'm curious :)

Can I ask why a 'single binary stream' view of a file would cause problems when dealing with something from a 'file is actually a bunch of records' system?

Like, if the FTP program sends an identical sequence of bits to the destination machine, how could the destination have problems deciphering the (identical) file?

I don't have any experience with VMS, but I'm assuming that there's information in the file (somehow, somewhere) that tells you where the records start and stop (whether it's a terminating character or a set size of each record, etc) because otherwise how would VMS know where to find each record. Is this not true? Is there information about the file that's stored outside the file?

As far as I remember when I used VMS while at the high school, if you write a text file in VMS and then read it byte-by-byte as binary, well, there are no line separators at all. Each line is seemingly a record in the file. I don't know more about the details, but the information is not in the byte stream of the file, it's probably some metadata elsewhere in the filesystem. Other OSs don't work that way. An FTP transfer in ASCII mode will insert line breaks where appropriate while downloading from VMS, but other file types relying on records can't be reproduced outside VMS as simply.

I am guessing that there is some interior "magic" sequence, similar to the UNIX magic in the "file" utility, that inserts some kind of padding.

That is a "wild assumptive guess."

Actually, Multinet FTP does have a couple of switches for "VMS mode," but SFTP2 got a working solution in front of us with less reading of the manuals.

POSIX was wise in having absolutely none of this. Would you rather Apple had won over this standard and we had resource forks everywhere? I understand that the largest use of alternate data streams at times in the past was malware, to evade virus scanners, which seems to be an anti-feature to me.

A file should be a stream of bytes that can at least survive transfer over a socket (that is 8-bit clean), unlike this ODS-2 detritus.

I don't care who wins - I don't have a horse in this race at all.

I'm sure the FTP devs were just trying to get something working - they weren't worried about supporting all the possibilities for the definition of "file".

I'd guess the only people who really care are those affected (mainly VMS/Apple people who have to send/receive files via FTP).

Well, I guess if I was trying to FTP a file that had data in an alternate data stream (ADS) from an NTFS drive, I'd care. But then I know that the ADS wouldn't be sent by FTP and package the file accordingly.

I don't think POSIX designed anything - they just codified/standardized what was running in the wild.

Wouldn't something like the VMS-equivalent of a tar archive of your files work better for round-tripping to a non-VMS computer?

The largest use of alternate data streams is actually when a user downloads a file from the Internet via Internet Explorer/Chrome/Firefox on WinXP+. The browser creates an alternate data stream, Zone.Identifier, for the downloaded file so Windows will know that the file may be tainted.

see https://guyrleech.wordpress.com/2018/08/06/where-did-that-do...

You know that e.g. linux supports extended attributes and they don't copy so easily over ftp either?

Yes, I know that ACLs, security contexts, and zfs/btrfs parameters aren't going to move. This would be know as "filesystem metadata," and the send/receive operations on zfs/btrfs would be the best bet to preserve them.

However, my ODS-2 problem was not metadata, it was data that lacked esoteric formatting. I certainly don't need that for Oracle RMAN - an FTP binary transfer will always works.

At some point you start wondering what's meta and what's data, after all.

Everything in the stat() structure is metadata. We could start there.

Why do we consider creation date to be metadata, for instance? In photos and audio files, it's so essencial that it is stored in the data stream, even though we call it metadata.

It's a fuzzy line and everyone is probably a bit discontent to where it's drawn.

Oh my, that prompted me to read Wikipedia - and we have MUMPS also from VMS. A quite different programming paradigm. The last common ancestor with current mainstream programming is really far back and it looks very odd.

New large scale multi billion dollar healthcare systems are still being written in it.

VMS was just recently ported to AMD64, so you might be able to keep using it if your vendor ports their code.

https://vmssoftware.com/about/v921/

> Thank heavens that UNIX thinks a file is a single stream of bytes. Windows continues the feature experimentation with NTFS "alternate data streams." My, how clever that is [not].

Macs use Resource Forks, a type of alternate data stream, yet I am frequently reminded (here) that macOS is Certified (r) UNIX (r)(c)(tm).

XFS was for media and big files. If you were on A/V production with huge recordings, XFS was the right choice over a network.

Oracle is putting a LOT of development effort into XFS, and I am assuming that they are doing so because it makes their database faster.

https://blogs.oracle.com/linux/post/xfs-2019-development-ret...

https://blogs.oracle.com/authors/darrick-wong

I have a habit of selecting XFS for all my Linux installations. It is a truly excellent filesystem.

Though I must confess the reason I initially chose it was because of the name.

The part about NTFS is slightly incorrect. It wasn't really a new file system. It was a copy of High Performance File System (HPFS) for OS/2, that was originally implemented in assembler by Microsoft's best programmer Gordon Letwin. NTFS and HPFS even used the same partition id.

The DEC people added some stuff.

"The MFT structure supports algorithms which minimize disk fragmentation... While this strongly resembles the W_FID of Files-11, other NTFS structures radically differ."

https://en.wikipedia.org/wiki/NTFS

"ODS-2 is the standard VMS file system, and remains the most common file system for system disks (the disk on which the operating system is installed)... NTFS - Has many structural and metadata similarities with Files-11 and is almost certainly conceptually derived from it."

https://en.wikipedia.org/wiki/Files-11

NTFS was definitely a new filesystem and supported many capabilities not in HPFS. Windows NT also shipped with HPFS support which was a complete reimplementation of the HPFS filesystem which shipped with OS/2.

They use the same partition id because that was the partition id used for IFSes in OS/2 (and therefore all non-DOS filesystems). This should not be taken as a sign of their similarities (or differences).

Oh thanks. I must have forgotten about that; if you have read the book, do you know if this was explained? Will need to read more and try to update the text!

Also some of the features listed (Compression, ACLs, volume sizes) were incrementally added over years and not available at first release, which somewhat invalidates the comparisons with other filesystems "Of the era".

The NTFS of today is rather different to the NTFS of 1993.

NT 3.1 absolutely did have ACLs in NTFS.

Sure, but were (relatively) limited compared to later releases.

And the on-disk format was completely changed for windows 2000, hence me saying what you call "NTFS" today is different to what it was originally.

If you differentiate between ext2/3/4, which are all forwards-compatible like NTFS, you need to do the same here. Though that "forward compatibility" may be an update pass when first mounting an older version of the filesystem, which may break backward-compatibility. Though that may be desirable, especially if you want to use the new features.

I remember there being a bit of a stink at the time when Windows2000 would automatically update the NTFS version and make it incompatible with NT4.0 if you didn't have the latest service pack, and even with that update some of the system tools didn't work on that filesystem anymore.

Is the book labelled as fiction?

I understand that early versions of NT emitted error messages identifying itself as OS/2. Did the book explain when NT became "brand new OS from scratch".

I think there's been a bit of revisionist history WRT the roots of NT. MS wants to make it something "we did without their help."

I didn't read your entire blog so apologies if you addressed this.

The NT kernel, a rewrite in C of the VMS kernel (which itself was written in assembler), presented three APIs to userland: Win32, POSIX, and OS/2. It seems reasonable that the OS/2 emulator/API/userland inside of early Windows would identify itself as truly OS/2, which it was likely seen to be internally.

The VMS kernel itself would also emulate programs written for the earlier RSX-11 operating system that ran on the PDP-11, so there is a history of emulation between these operating systems.

"Broad software compatibility was initially achieved with support for several API 'personalities', including Windows API, POSIX, and OS/2 APIs – the latter two were phased out starting with Windows XP...

"Mark Russinovich wrote in 1998 that while creating Windows NT, Microsoft developers rewrote VMS in C. Although they added the Win32 API, NTFS file system, GUI, and backwards compatibility with DOS, OS/2, and Win16, DEC engineers almost immediately noticed the two operating systems' internal similarities; parts of VAX/VMS Internals and Data Structures, published by Digital Press, accurately describe Windows NT internals using VMS terms. Instead of a lawsuit, Microsoft agreed to pay DEC $65–100 million, help market VMS, train Digital personnel on Windows NT, and continue Windows NT support for the DEC Alpha."

https://en.wikipedia.org/wiki/Windows_NT

Information on the Windows POSIX subsystem:

https://en.wikipedia.org/wiki/Microsoft_POSIX_subsystem

"The NT kernel, a rewrite in C of the VMS kernel (which itself was written in assembler), presented three APIs to userland: Win32, POSIX, and OS/2."

Not quite. I'd rephrase this as "NT presented three userland personalities—the CSR (Win32), OS/2, and POSIX—through a composition of kernel primitives implemented as userspace subsystems."

The core binary image ntoskrnl.exe only exposes the primitives needed to get the CSR, OS2, and PSX subsystems started, and those (mostly, save for some code that lives in win32k.sys) run in userspace. They more or less act as translators between the native NT ABI/API and the respective subsystems' ABIs/APIs.

PSX in particular did a lot of its own PE image mapping and parsing (something that you'd perhaps think would be done in kernel space) so that it could defer to the other subsystems to run foreign code among other things.

I think my quote of Mark Russinovich is a reasonable basis for claiming that that the NT kernel was the VMS kernel rewritten in C.

You can edit the wiki if you strongly feel this not to be the case, but I have seen this claim in many other places.

You'll note that I didn't address the VMS rewrite at all.

True, and I will upvote you for my oversight.

The book doesn't shy away from describing the collaboration between Microsoft and IBM. Also, the book depicts Microsoft knowingly screwing over IBM in their collaboration.

So Gates sided with Maritz. NT would assume the Windows “personality,” by assuming those of its attributes that were visible to the customer. NT also would be redesigned to run applications written specifically for Windows. But Gates added a wrinkle to his position. Given the depth of IBM’s attachment to OS/2, Gates reasoned, abandoning the program altogether would only trigger a nasty split with IBM. Why not maintain a pretense of cooperation by convincing IBM of the merits of an operating system that hosted both Windows and OS/2? NT could be designed to do this, Gates reasoned.

If IBM accepted this, Microsoft could continue to describe Cutler’s program as “NT OS/2.” Even though it contradicted his rhetoric about the need for a single software standard, Gates was prepared to tell IBM: “There’s no reason to think two [program personalities] can’t be of equal importance” to customers.

This was a bold stroke. By denying reality, Gates’s tactic would freeze IBM long enough for Microsoft to install Windows as the standard for computing. Gates would peddle this apparent compromise to IBM and rival software makers, while Maritz and Cutler’s team pursued the real objective internally.

...

Cutler asked about Ballmer’s promise that NT would still sport an OS/2 personality. Not to worry, Ballmer assured him, someone else could take care of that. Indeed, OS/2 was now so unimportant to Microsoft that Ballmer had assigned a tiny team in Israel to create an OS/2 variant of NT. The Microsofties hated OS/2 so much they couldn’t even bear to have someone working on it in the same continent.

Dave Cutler has elaborated on a lot of this stuff in some recent YT interviews that are excellent: https://www.youtube.com/watch?v=9K3eMzF6x28

Part of it is Cutler really really hated working with IBM... it starts around 3:00 minute mark.

Several MSFT employees I talked to mentioned that they didn't like working with IBM - too much of a culture clash from what I could gather.

An earlier comment mentioned Gordon Letwin[0] (lead architect of OS/2 at MSFT), and that led me down the rabbit hole to an "infamous" USENET post he made re: OS/2[1].

It sounds like there was acrimony all around:

Another problem that came up here was that IBM didn't want us to use the windows API for the graphical environment under OS/2. Many key folks inside IBM had always hated Windows. IBM had this crazy thing called TopView, it was a character oriented windowing scheme and not very good. Bill Gates, myself, and some other folks made several trips to Boca Raton to try to explain to those guys why a character oriented windowing scheme was obsolete before it was even written, but to no avail. One of IBM's most major problems is that although their top guys may be smart, they aren't techically savvy.

[0] https://en.wikipedia.org/wiki/Gordon_Letwin

[1] http://gunkies.org/wiki/Gordon_Letwin_OS/2_usenet_post

You cut off the best bits of Letwin's rant:

"...although their top guys may be smart, they aren't techically savvy. And their low level guys are often neither. IBM doesn't promote on the basis of your skills and ability; they promote on the basis of seniority and other secondary factors. So the guy who makes these decisions often doesn't know what he's doing. And he doesn't know that he doesn't know, because his peers are equally butt-ignorant too. So these guys can never figure out how other folks, including but not limited to Microsoft, keep doing better! Must be dumb luck, they think. I always agreed that it was dumb luck. If you catch my drift... :-) "

> Is the book labelled as fiction?

> I understand that early versions of NT emitted error messages identifying itself as OS/2. Did the book explain when NT became "brand new OS from scratch".

Windows NT started out as Microsoft's effort to do OS/2 NT. There was a 32-bit Microsoft OS/2 2.0 (distinct from IBM OS/2 2.0), and OS/2 NT was to be OS/2 3.0.

OS/2 NT wasn't rebranded Windows NT until Windows 3.0 eliminated OS/2 from relevance, and with it, the Microsoft/IBM partnership. That's when IBM introduced their own (very nice) OS/2 2.0 and started a years long effort to eliminate Microsoft code from the OS/2 codebase (and the Microsoft copyright message from all over the UI)

It's been a few decades since I read the book, but note it was published just after NT was released, when everyone knew that it had been called OS/2-NT for a while. So probably no wool pulling going on there. Note also that this was a naming move. NT was never based on OS/2. otoh in my experience (I've worked for two companies that had books written about them), book authors tend to a) focus on drama and b) get their information from a small number of senior participants.

My understanding is that the original collaboration between Microsoft and IBM was around certain parts of OS/2, not the whole system, so it is plausible that Microsoft worked simultaneously on OS/2 and NT as the book describes. Note that NT is heavily based on the work previously done at Digital on VMS (by people from the same team), so there are some pre-existing foundations there, at least in terms of design principles and ideas.

https://en.wikipedia.org/wiki/OS/2 provides a recap of the history but I cannot find details on exactly who did what in that page. And my memory from what I read in the book is fuzzy at this point...

Reply on news.ycombinator.com