I agree in principle, although exactly who counts as a "visible minority" is a bit of a complex question.
In Sydney, Australia, a woman wearing a niqab is still a rare sight, and she is likely to get starred at, a lot–even other Muslims are going to stare at her. Women wearing hijabs are a far more common sight, so they will get starred at a lot less, although I'm sure they will report being starred at sometimes too. But, a lot of Muslim women don't wear any head covering – like the women in my uncle's family – indeed, in Albanian Muslim culture, women rarely cover their heads, and a woman who starts wearing a hijab is likely to be suspected of religious fundamentalism, which is viewed about as negatively in their culture as in my own. Should my uncle's family be considered a "visible minority" then?
Sydney is over 20% Asian (including East Asian, Southeast Asian, and South Asian people in that term)–Asian Australians are a minority, and obviously a visible one–but do Asian people get starred at in Sydney solely on account of their race? You usually stare at the unusual, but in a city which is over 20% Asian, there is nothing unusual about seeing a Chinese/Indian/Japanese/Korean/Vietnamese/Filipino/etc person, you see them every day, you count them among your friends, classmates, coworkers, neighbours, etc.
I remember one time I ventured with a Finnish colleague to a working class bar in Bangalore, and we got a lot of stares there, because nobody was expecting to see people like us in such an establishment – by contrast, in the airport, hotel, offices, shopping centres, tourist sites, etc, nobody starred at us, because Indians are used to encountering foreigners in those places.