In my case, people often tell me that I have a very clear British accent; but when I ask them to pin down what sounds so particularly British, I often flounder about and eventually answer along the lines of "well, you speak very precisely and in a very educated manner" -- which says something about our (or at least, Canadians') stereotypes about the British!
In light of this, I wonder to what extent the judgements made about whether people "sounded black" or "sounded southern" were in fact merely that people were picking up voices which sounded uneducated -- and having subconscious racism translate that into a sense of blackness or southernness.