The important thing being conveyed is lack of diversity, not whiteness. In the United States, saying something is "mostly white people" conveys that other ethnic groups are not present for some reason (whether it be them self-selecting to not participate or because of exclusion). The purpose is almost always to note diversity or lack-thereof, and assume that the person is making a value judgement about the people because their skin is white is to completely miss the point.
For example, if you bought a pack of starbust candies, and they were almost all orange, you might note or complain that they were almost all orange. I would do so, and orange is my favorite flavor of those. The issue is, I desire the diversity of colors and flavors, and too much orange is not as good as what the added variation brings.
So, "mostly old white people" as a negative doesn't mean old people are bad, or white people are bad, or old white people are bad, but that mostly old white people are not as good as a group that has more variation in it. "Mostly" is the word you should be focusing on, not the other ones.