I admit I'm uncomfortable explaining it by skin color, but I've never found anybody who has been able to explain to me what the difference is, and I will admit lacking the musical knowledge to explain it. I fell into that style from dancing Lindy Hop; I was loving the music I was hearing when I was out dancing, so I went and bought an album from the only name I knew at the time - Glenn Miller. It was some of the most boring and trite music I'd ever heard in my life, and did not inspire me to dance.
For me (and I don't judge people for thinking differently) there's a certain joie de vivre in the music that is just lacking from what white musicians released commercially. I know they were capable of it (I once found a recording of Glenn Miller swinging it just as hard as anything Basie put out) but they were playing to their audience at the time. As I've learned more about the history of Swing and Lindy Hop, this was a specific choice made to "civilize" (as the white people of the time would have said) Jazz's savage rhythms. There's actually posters from the Arthur Murray school in the 1940s saying this exact thing.
(Aside, I had a friend who played a radio show in the 2000s playing old Jazz music. She told me once that if she put anything from a black musician on the show, she'd get hate mail from the listeners. Go figure.)