Power inbalance man, that's the point.
If I'm a rich man and I ask a poor woman out, her answer actually might have a much bigger impact on her economic well-being than if I were an employer interviewing her for a job.
Does that make it wrong for rich men to ask poor women out?