No, the "best people" weren't responsible for slavery. Any rich guy can write a book or do math especially if they live off slave labor.
There were plenty of people who opposed slavery. Those were the best people, even if they're not remembered in a culture that worships slavemasters.