The big problem is that American women are prone to "groupthink", and judge a man's desirability based on how attractive they presume other women to find him. So a guy with a low number is perceived as unwanted for that reason alone. Women in the rest of the world are a lot more independent-minded and don't fall to this.
I take it the other way. I've decided that if the majority of women in NYC aren't going to respect me because my number is (voluntarily) low-- not zero, but single-digits-- I shouldn't feel bad about being an unforgiving bastard when it comes to women being too experienced/"stretched out". Regardless of age, I don't think I could marry a woman whose number's higher than 5.