Quiet NaN is critically different from the Maybe monad's Nothing value. Maybe gives you proper exception semantics even if it's implemented by means of predicated data flow rather than early-out control flow (and in a pure, non-strict language like Haskell there isn't a real distinction between data flow and control flow). Whereas quiet NaNs have a significant issue with non-termination where you have to very carefully express all your loop conditions positively. For example, 'while (x - y > eps)' will terminate if either x, y or eps are NaN, but any code that relies on 'if (positive_condition) break' and similar constructs for termination will loop or recurse forever if the condition operands become NaN.

A closer analogue of NaN might be the x ("unknown") value in the three/four-valued logic of hardware simulation languages like Verilog. But the situation in Verilog, despite its challenges, is so much better since you don't have a hard discontinuity between "numbers" and "logical values". Numerical comparisons between unknowns can produce unknown logical values, and you have equations like true & unknown = unknown, true | unknown = true, ~unknown = unknown, etc, so you don't have the issue with NaNs where as soon as something crosses from the floating-point realm to the integer or logical realm, it has to commit to a definite value, with all the aforementioned issues.

All that said, NaNs were the right design choice given the constraints. But it's far from ideal.