As long as the hardware isn't DMA-capable or has to be shared, allowing programs direct access
isn't a security violation. Since everything runs in protected mode, the kernel is still "in charge", and can limit what I/O should go directly to the hardware. This is supported by the processor, which has a per-task bitmap of allowed I/O ports.
Virtualization was introduced on IBM mainframes in the 1970s, and in a somewhat more limited version has been present in 32-bit x86 operating systems (even Windows 3.x could run several virtual DOS machines at the same time).
Every 32-bit version of Windows still supports running 16-bit programs. Microsoft could have continued supporting 16-bit protected mode even on 64-bit, or even used the newer virtualization features (in place of V86 mode) to run real mode code, but choose not to.
As for the expectations of modern computer users, those that don't know how any of this stuff works probably believe that everything has to get slower, more restrictive, and less backward compatible in the name of Security :(