Even if you want to argue that the reasons people have for deciding not to get involved in open source are wrong or that those intangibles shouldn't affect open source projects for some other reason, the fact is that the pattern will nevertheless self-reinforce and affect open source by deterring more hands from coming on deck.
If x% of the white male population is developing on open source, but only x% * y% of the non-white-male population is, and getting people interested in open source isn't a zero-sum game, then making y as close to 100 as possible can only be a good thing. Making a concerted effort to recruit diversely, it is hoped, will have accelerating returns as intentional involvement of diverse contributors will lead to more and more organic involvement of diverse contributors.