Beyond that, with a few exceptions, most of the signal rolled off by the switch is stuff you don't want to hear anyways - "plosive" pops, rumble from low-frequency buildup in rooms, vibration, etc. Much of it can't even be reproduced by most speakers (speakers don't do low bass because, as said before, it's a hard problem and not actually audible in most cases).
As an aside, when mixing records, I'll often roll off guitars at 200 or 300hz, losing a couple of octaves of low-frequency information. In theory, this is awful. And if you listen to the solo guitar tracks, it's awful. But in an actual mix, it's great! It gets the low end of the guitars out of the way of real low-frequency instruments like bass and kick drums, making the bass sound tighter and cleaner. The guitars sound clearer too, because they aren't getting muddied by competing bass instruments.