I concur, macOS is a nightmare. I didn't think so before I had to write code for it, but I am currently writing code that runs on 5 OSs (Linux, Windows, macOS, Android and iOS) and macOS is by far the worst of all. In particular, everything related to filesystems was a nightmare, especially HFS+. Thankfully Apple is replacing it with the much more sane APFS.
I don't have much experience with them but I think VxWorks and QNX are good examples of simpler OSs. I do have some experience with Minix3 and it is certainly one. I guess the BSDs stand somewhere in the middle.