This condescending comment ("whatever negative things happen to you", "more complicated than that, dear") ignores at decades of research in sociology, critical theory and political philosophy - which, at the very least, say that concerns about the effects of gender in the structure of society cannot be so easily dismissed.
>but rather with one single factor-- some ethereal 00.00001% group of conspiratorial, ostensibly straight white old men
This is not what "patriarchy" or "the patriarchy" refers to in any academic text. What you're saying is more akin to an anti-male conspiracy theory. The claim is not that there is a group of conspiratorial men in charge, or even men with bad intentions. You can read the most rudimentary details of the theory at Wikipedia[0] or at the highly regarded Stanford Encyclopedia of Philosophy[1].
I'm not a strict adherent to the theory myself, but I think it's worth pointing out when a claim as to what the word means and what the theorists say is obviously wrong.
(Side note: it's curious how my comment, in line with current research and citing respected sources, is downvoted by people who don't bother to engage with the arguments themselves.)
[0] https://en.wikipedia.org/wiki/Patriarchy
[1] https://plato.stanford.edu/entries/feminist-power/