Well some have tried. There was a civil war about it.
In its simplest form, the civil war wasn't fought because the north thought slavery was bad. It was fought because several states tried to leave the United States.
Hint: Most of the secession declarations include language like this:
> But an increasing hostility on the part of the non-slaveholding States to the institution of slavery, has led to a disregard of their obligations, and the laws of the General Government have ceased to effect the objects of the Constitution.
Citation needed.