Without a doubt it's racist. But if you consider power structures and privilege, not all racism is equally bad. As a white male, I'd rather he not have that opinion of me based solely on my race, but I'm willing to give him some leeway due to his experiences.