I strenuously agree with your overarching point. But I have a more charitable perspective about the accusations of misogyny. I think that many, many women have had awful and in many cases traumatizing experiences in the world of tech. A lot of people associate opinions like Damore's with the bad behavior they have seen and experienced. They see these kinds of arguments as being part of the same phenomenon; men who hold opinions like these are the same kind of men who would talk over women in meetings, dismiss the achievements of women, be less likely to promote women, pay them less, harass them, etc.
I think a lot of people (women especially) felt an extreme and visceral reaction to the memo, because they intuitively made this association that these sentiments were related to their bad experiences. And while I strongly believe that the accusations of misogyny were unfair and misguided, I interpret them charitably as an attempt to push back against what they interpret as sexist behavior. They are trying to support the women around them.