This is the argument conservatives made in the 1950s and 1960s against blacks having the right to vote, drink from water fountains, swim in public swimming pools, sit anywhere on a bus etc. They said they were not against blacks per se, but against federal encroachment, communists in the civil rights movement etc.