This line of reasoning is pretty poor. You don't need to be someone's boss in order to have a huge ability to harm someone's life. All of us are capable of doing great harm to others - we just choose not to.

Saying "she can't consent because he could harm her" is like saying I can't deny giving my neighbor some milk because if I upset him he has the power to break all of the windows on my house. Since when do we focus on these hypotheticals? Since when do we always assume malice? We trust that people are good, prosecute the bad ones (both breaking windows and firing over sexual denial are illegal), and allow neighbors to ask for milk because 99% of the time no windows get broken in the event we say no.

Sure, the analogy might be a bit contrived. But I think it still does a good job of demonstrating how ridiculous this line of reasoning when applied outside of the sexual harassment thought bubble.