The effect is massively amplified if you give women drugs because they don't have testosterone levels like men do. Those records that were set in the 70's pre-testing will likely never be broken.
The issue is that if it is legal, there is a huge incentive to take it to an abusive point especially in a celebrity economy like what exists is professional sports.
In some sports, professional athletes are almost constantly injured. Off the top of my head I can give you MMA (UFC) and american football as examples but I'm sure there are many others.
Healthy sports end where competitive sport begins.[1]
Right now we have an arms race that barely keeps it at bay. It's not an ideal solution but I think its better than athletes fully abusing their bodies with massive amounts of drugs like they did back in the East German Olympic days.
For non-professional athletes, I agree with you. It should not be demonized to the extent it is.
1. Quote from Brecht, referenced many times, e.g here http://gladwell.com/drugstore-athlete/