I would hesitate to call Gandhi a racist, but he definitely was prejudiced against black Africans (in the beginning) as a result of what would today be called brainwashing / indoctrination (of Macaulayism) by the colonial British. Due to this policy of Macaulayism (to tolerate certain indians educated in the British system as equals), Gandhi was largely insulated from British racism in India and UK. So he had not only inculcated the idea that western culture was superior but was also a strong and vocal advocate for the British way.
What changed his worldview was the racism he experienced in Africa. In general, indians were treated differently in Africa. But he also personally experienced racism when he was once asked to get out of a first class compartment, despite having the train ticket for it, because a white man had demanded that he be removed.
The other major thing that changed his opinion of the British completely was when he saw the brutality of colonialism and war. A naive Gandhi believed it was his duty to assist the British empire and so he volunteered as a non-combatant with other indians, to assist the British effort in the Anglo-Boer War.
What he saw horrified and scarred him for life. He then swung to opposite spectrum and believed the British to be the devil's incarnation.
He was in Africa for 21 years and his life experiences, and African culture and politics, greatly influenced his political philosophy. As his political philosophy evolved, he found the right balance between blind admiration to blind hatred of the British empire, their culture and indian culture too.
The fact that we know about it means that it did register.
Consider a person of colour living in the eponymous mansion of the city's wealthiest neighborhood.
For a racist government, it's a symbol of defiance.
For other people of colour, who have been told by the government their whole lives that they are worthless, good only for manual labour -- one of their own is living in the oldest mansion in the fanciest neighborhood in town. It's an affirmation of their worth.
As a symbol, his place of residence had great power.