I don't see why Linux couldn't do the same; open /sys/kernel/something and epoll on it.
During this - unlike Linux - you can actually use the mouse, CLI and close programs yourself.
On top of that server applications like IIS has built-in watchdogs. If an IIS process grows to use too much memory (60% IIRC) or excessive CPU, the watchdog will recycle the process.
1. Show a GUI with a choice
2. Show a message on the current terminal and ask what to do
3. Just return "kill it now" if there is no interactive session
And if there is no such service, just default to 3. The problem really is that the state cannot be captured and communicated to the user. I doubt the NT kernel itself shows a GUI window, it's probably a service that gets woken up by a kernel exception, which in turn shows the window. Basically, the Linux kernel needs more pluggable functionality for user interactions. It's absolutely fine and even recommended to not have an entire GUI in the kernel, it needs to just provide a mechanism for userspace to capture the event and decide what to do with it.
Catching an otherwise-fatal out-of-memory fault and recovering would be too complicated / bug-prone.
Then again, Android has a customized Linux kernel.