Problem isn't just the fear of rejection. There are legitimate reasons why some people get rejected more than others, whether it's looks, social skills or just lack of shared interests.
If you are part of those people, the "return on investment" in terms of socialisation diminishes - it's not about fear per-se, it's about the rational decision of not sinking time into an endeavour with a high failure rate if you have other, better options.