While I agree with you in principle, I don't in practice.
The whole problem is that TV remotes keep changing. My TV now we have two remotes, one that came with the TV and one for Google TV.
I used the TV remote the other day out of necessity...and it took me about 20 seconds to find the mute key. On the GTV remote, it's one of like 5 buttons.
I'll always prefer physical buttons to menus, but not when the layout and meanings of said buttons change every year.