Why should it matter that they're physically indistinguishable? Would it be more excusable then?

Caste is just another uneducated, thoughtless distinction to discriminate against other people.

Caste is as correlated with cultural identity and upbringing in India as religion, nationality, and skin color are in other parts of the world.