This was much the same criticism leveled against Donald Trump. Because he did not clear denounce the racist activities in Charlottesville, he could be interpreted as implying support.
Personally, I am not seeing the lack of things that are actually sexist in Damore's memo. His one-sided and dishonest interpretations of his sources alone make his sexism clear and it's the same kind of dishonest interpretations we see from the racists and the sexists.