The usual argument against clothing restrictions (see also supershoes in running and various aero stuff in cycling) is that you want the sport to reward the best athletes rather than turning into a technological arms race. This is especially complicated in sports where people don't get to choose their own gear and so (for instance), whether you have access to the best shoes depends on who your sponsor is. Back when Nike was first rolling out the first supershoes, you would sometimes see athletes sponsored by other brands actually wear Nikes with the logo blacked out, because it was just such a big advantage.
As another comparison point, look at Formula 1, where technology is a huge part of the competition, with the result that a driver can be dominant one year and then fall way back the next because of some technological shift. Of course, even F1 does tinker with the rules a lot to try to preserve competition, as when they banned electronic stabilization.