I wish I had some authoritative references for this. Hopefully somebody else can write authoritatively or correct me. Based on my readings, slavery was always a problematic aspect of America.

Even the founders debated whether to abolish slavery, but ultimately punted because they felt uniting the states was more important.

That is, it was a tough compromise.

Take a look at the book The Long March (about Lincoln). There was a several-decade long movement before 1864 to abolish slavery.