I have many white male coworkers (as you'd expect given the demographic statistics for tech) and I have nothing but the utmost respect for their ability. I don't harbor any assumption that, "that they've had a silver spoon in their mouths their whole life."
I don't think that believing that there are groups that have been more economically and socially marginalized than white males is alienating. I also don't think that it implies that I believe my white male coworkers have been given a free pass to the positions they are in today. Everyone I work with has had to pass the same bar.
As I've stated, all of the statements I've made here are my own, made on my own time, on a public site and do not represent the views of my employer. I leave my politics at home.