Accusing people of not reading the memo doesn’t contribute anything to the conversation. If you share Damore’s views, just say so rather than trying to distract.
The other is that is it often given such uncharitable descriptions that people who have read it simply can't believe the hate levied at it comes from actually having read the thing. I've actually talked this whole topic through for hours back and forth with someone who is very against the memo, but even they they would call it a sexist manifesto. That sort of thing comes across as off the cuff criticism by people who haven't read the thing.
If he's made direct statements rejecting biological determinism it should be easy enough to cite them — and it'd be useful to do so to avoid distraction when his supporters try to rehash the arguments in favor of that position.