It's surprising to you because North Americans have gotten used to saying "people of color" as if they were all united.
Upper-caste Indians basically see themselves as people on top of the racial hierarchy. In many ways they identify far more strongly with British or American elites than with Dalits.