I mean, there's not much high-brow about Cole's music, whereas Dylan has tons of academic studies on his "poetry" (whether one consider it actual poetry compared to people like e.g. Wallace Stevens and e.e. cummings or not). With Cole (and whites like Bing Crobsy, or band leaders like Glenn Miller etc) it's mostly entertainment, not really culturally significant. Sinatra goes a little further, but again, not that much. Compared to them Dylan is like a literary titan, and more adventurous musically too.
I'm not sure about whitening it out either. Before Dylan there was Elvis, Jerry Lee Lewis, Roy Orbison, Buddy Holly etc who have already whitened the scene. And of course whites do have a right to have their own music (like the folk/irish/country/rock etc thing Dylan combined) heard.
Is Cole really "blacker" than Bing Crosby?
Does Cole really represent black culture as a singer, or more what the while mass audiences wanted to here at the time? How many of his songs are written by white songwriting teams?
Now, if Dylan had replaced Mingus, Billie Halliday, Robert Johnson, et co, then, yeah, that could have a basis to speak of "whitening" and "dumping down". But not for Cole et co.