Yes, this is certainly more "far-reaching" than other env vars. But If you have the ability to wreak havoc even if ld-linux wouldn't check for this (obv. PATH being the prime example, but unsetting HOME will almost certainly break some software written by novices (not falling back to the home specified in /etc/passwd)).

And I also could imagine that tools that allow setting arbitrary env vars might filter common troublemakers like PATH, but almost certainly not niche env vars that change the bahviour of ld-linux.