Also RTLinux kernel: https://en.wikipedia.org/wiki/RTLinux
In this model, your compositor, input pipeline and (at least the UI latency sensitive part of) your applications would have to be ported to the RTOS, which makes this pretty infeasible. But it works really well if you have a some hard-realtime control loop talking to a bunch of non-realtime network IO or UI code. Would be a fun way to build a wifi controlled quadcopter.
These days you probably want to use Xenomai, or possibly RTAI.
Thanks for the great writeup correcting me!