If my boss says, "Hey deong, what do you think about black people?", that's not a get-out-of-jail-free card for me to start dropping n-bombs. There's one answer you're allowed to give to that question, and it's not because there's some conspiracy to oppress dissenting opinions. It's because there are extremely reasonable standards for employment that include not shouting racial epithets and white supremacist propaganda. Being asked your opinion doesn't change the rules.
Damore was certainly less inflammatory than that, but the point remains. You can argue that his opinions didn't warrant firing, but there's no reasonable way to argue that because he was responding to requests for comment, it's unethical or otherwise shady to hold him accountable for those comments.