You seem to be missing important parts of US history, not least of all the century of Jim Crow. I mean, come on, my Boomer parents were already alive to see the Civil Rights Act passed.
For a trace of the thread directly from the civil war to the current Supreme Court, check out this podcast episode: