All this for the possibility of interposition?
Yes, it seems so. The impact is there for this little-known and little-used feature. Instead of optimising for the common-case scenario where the symbols are not overridden, the ABI optimises for the corner case.
Using LD_PRELOAD isn't so rare and strange that you can propose throwing it out without quantifying its performance cost. I can't recall offhand why I needed it (I would guess Valgrind or Massif) but I've used it several times as a developer. What exactly is the payoff for giving it up? It can't be bigger than the current performance difference between statically and dynamically linked executables, can it?