I suggest you read this:
https://www.overcomingbias.com/2018/06/comparing-income-sex-...
https://www.overcomingbias.com/2018/06/comparing-income-sex-...
From what I can discern, he makes two counter arguments:
1. Sex is as important as income.
2. Poor income leads to "poor access" to sex. If the income distribution is unfair, so must the resulting sex distribution be.
To 1 I can only respond that many people feel differently. Anecdotally, I've never encountered anyone but well off men making the argument. Is that your experience also?
For 2: Poor income leads to many unfair situations. Let's cure the problem and not the symptom.