At first I misunderstood the title, thinking that for your hobby OS you targeted gameboy hardware. Makes me wonder what would be the first game console you could really make a usable GUI OS on.
Moving up to the Sega Genesis with its 16-bit 68000 that many old UNIX systems were built around: it still only has 64 KiB. But, hey, that’s a lot better than 2 KiB, and it has memory protection. But the 68000 can’t do virtual memory (unassisted) and I don’t believe the Genesis has a timer interrupt, which seems like it’d complicate multitasking (I guess you could run the scheduler when the graphics system raises the vertical retrace interrupt?) but IMO it’s about the minimum level that could run something like what we tend to think of today when we talk about an operating system.