The problem with set-user-ID executables is that they (and the dynamic loader they use) effectively operate in a "blacklist" basis: there's a list of "evil" things they have to protect against (in this case, environment variables controlling the dynamic loader), and if they fail to protect against one of them (in this case, the dynamic loader didn't think of protecting against low RLIMIT_DATA, though a component called later by chpass did) it's game over.

Contrast with something like a service called over dbus: the caller has little to no control over the service's environment, and the most the caller can do is try to confuse it with nonsense dbus calls. It's a "whitelist" basis: only calls the service understands (or think it understands) can do anything, everything else is rejected by default.