(Can I legally port, say, the GPLv2 ext4 filesystem driver from Linux to a non-GPL OS like OS X? Can Apple pick up that port, and distribute ext4.kext and its source in OS X, without putting the entire xnu kernel under GPLv2?)
The harder and fuzzier part is that the ESX boot process works by asking Linux to "load" vmkernel, using the vmklinux helper module. If this is in the sense of loading a kernel module, it's (mostly) clearly wrong. If this is in the sense of loading a userspace app (as if vmklinux was a binfmt module, and vmkernel was a non-ELF executable), then this is (mostly) clearly okay. But it's doing neither of those. It's loading vmkernel above Linux, making the Linux process the first world, analogous to a Xen domU. So this isn't userspace, but it's also arguably a new address space. Neither of which are legal concepts per se, so it's hard to tell where this lies.
This fuzzy part is neither the first component (Linux / "vmnix" / the bootstrap) nor the second (vmkernel), but the way in which they interact and are designed around each other. This itself might be an infringement, even if the first is not infringing, and the second is fixed.
(Is the word "vmnix" wrong? Does it apply only to the entire Red Hat-ish distribution in the COS, and not to the Linux stepping stone alone?)
Basically the old 1.0 - 4.0 process for _ESX_ was like this:
* Bootloader
* Linux Kernel boot
* Load VMnix module.
* Load VMkernel
* hardware control passed to VMkernel
* Linux continues running as a sort of 'super' VM with some limited hardware access.
For _ESXi_ from 4.0 - now :
* Bootloader
* VMkernel
I don't know if this was to make the transition for stuff like vpxa from ESX 4 on easier internally, or to make it easier for 3rd parties to develop against, or both.
Is it lazily wrapping some functionality that should be open sourced that fills in where the service console would normally be? Probably not, but I can see why people would be a little suspicious.
vmnix was the name of the Linux kernel module that was loaded during the Service Console's boot process.
The terms "Service Console", "Console OS (COS)" were at times used interchangeably and referred to the Redhat distro you logged into at the terminal.
VSphere 4.0 had 2 versions: ESX (with COS) and ESXi (without COS). In 5.0 the COS was removed entirely.