It's not just "association" as you put it. r/the_donald was singing is his praises. They're a pretty toxic bunch and I've found that I have very little in common with many of them (mostly because I find their views to be <insert any negative adjective>). So when they're the only people I see coming to someone's defense I assume that person said or did something pretty bad.
Mike Cernovich is the same way. He's looking to push a narrative to further his own agenda. An agenda and narrative I find gross and offensive. So he thinks Damore fits into that narrative. That is a not so subtle hint to me that Damore is saying something gross and offensive.
It's not the best way of judging things for sure but I don't think it served me poorly here.