Most bootdisk-based games did not have any kind of "customized version of an operating system". They were running on bare metal. For 99% of the games, there was no abstraction/management layer between the application (the game) and the hardware nor any kind of framework that would impose a certain structure on your application. Of course, game code was structured and you could often find subroutines for recurring tasks (clear screen, wait 100ms, read track from floppy,...) but even those were sometimes directly inlined in higher-level code (e.g. game AI) where it gave a performance advantage.
Only complex games had some kind of very thin abstraction/management layer that you could call an OS (often MUCH thinner than embedded micro-OS like RIOT or Contiki). As an example, Carrier Command had a quite general GUI system.