Yeah, I've heard this theory before, and it kinda makes sense. Except that there are other countries (most of the Anglosphere, including the UK) that also had slavery, (and/or have confronting relationships with their indigenous population), and don't feel that same fear.
Is there something unique about the USA's history with slavery that would cause this?