Bigger attack surface in the kernel, for something that doesn't need to be there, and that is historically very difficult to code securely.

grsecurity just disables it outright IIRC (the new one that was added sort of recently, probably year+ at this point).