That's absolute bullshit. That's "those people" kind of talk, lumping everyone who shares one characteristic into one group and dismissing their rights as meaningless because they aren't fighting for your preferred cause.

Among the most intersectional activists I know are queer people of color. They're the folks standing up for immigrant families in detention (which is a quite lonely activist area) at a much higher percentage than the population at large, for example. It's certainly not an issue I see white straight male libertarians get up about (I fit that description in many regards, but I'm frequently embarrassed by how self-involved and self-serving the so-called "liberty" movement is).