That's more than just device-drivers, you have to work out a consistent interface between the 'system' and all of the peripherals, rather than looking at the individual interactions of the system with individual peripherals.
In a way, you need something like the UNIX concept of 'everything is a file'. (Only more so.)
My hobby is writing emulators for real machines from the past. Most of my effort is in emulating the peripherals interactions with the CPU and RAM.
I have recently been looking at minicomputers of the 1970s. They seem to have had the concept of the peripheral doing all the work, with the 'system' merely 'asking a limited number of code questions' and the system 'getting a limited number of answers' from the peripherals.