As a trivial example, what happens to quadriplegics? Do they not get to go to MIT?
Also, you can fulfill the swim requirement by taking 1 beginner swim class regardless of whether you actually learn to swim during that class or not.
Rather a lot of the history of disabled and otherwise 'unusual' people seems to disprove how 'obvious' and 'commonsense' that idea is in practice.
Furthermore, where is the line between disability, principle and preference? Do religious reasons count? What about cultural ones? A friend of mine who was a Navy officer told me that the Nepalese he trained were absolutely terrified of water - the rivers were so cold in Nepal that they had a deep-rooted cultural association of water with danger. They needed to persevere because they had enlisted as frogmen, but why should a student of astrophysics do so?
The opportunity to justify one's reasons doesn't absolve the issue entirely either, as that effectively makes skills of persuasion, rather than swimming, the requirement. Finally, if you allow arbitrary requirements, which ones are reasonable? Swimming might be as unreasonable to someone brought up in a desert as mountain hiking might be to someone born and bred in Massachusetts.
I have zero issue with forcing prospective astrophysics students outside of their comfort zone.
ADDED: There are also phys ed requirements and humanities distribution requirements. If these are such issues, I have no doubt there are plenty of other schools that don't impose them.
Still technically a "requirement" though.