Heck, you can get away with that today, by just creating a virtualization platform like Xen, but with much more carefully-thought-out exposed virtio hypercalls; and then writing unikernels against that interface.
Though, from another perspective, this model is exactly what the "transitional" era of game consoles did. The Nintendo Wii U, for example, normally [i.e. during a game] has a unikernel running across two "application processor" cores, that sees those as the only cores and has direct ownership of the peripherals; but then the OS [running on a third, separate, low-power core that the game VM has no knowledge of] can, at any time, force the game VM into the background with an interrupt, whereupon it must relinquish one of its two cores, some memory, and not touch the peripherals at all until the OS says it's okay; and then the OS can use the borrowed high-power application core, memory, and peripheral-hardware ownership to temporarily foreground an OS modal GUI, like a game manual or a web browser or an instant message.
Sadly, this was all abstracted, for the game developer, behind an HLL [that is, C] SDK—which is sadly a "requirement" in the modern era, given that devs are going to be porting their games to/from PCs.