It's not only social anxiety that forbids people. It's also failed relationships that bruise your desire to try again. In my case I failed people enough time not to want to bother someone else :)
I roll my eyes when I hear quotes like, "Judge a fish by it's ability to fly and you'll think it's a failure"... but there is something to that.
Maybe you're chocolate and those people just prefer vanilla?