It's also important to remember that this was before the industrial revolution, when almost the entire economy was agricultural, thus the South was far more powerful than the north.
The Civil War started right after the industrial revolution ended. By this time, the balance of power was greatly in favor of the North.