There is truth to it. A lot of anti slavery but still racist northern whites thought slavery was bad for southern whites. Thought it bred laziness in them and didn't incentivize industrialization.