Read about the Bloods and Crips in the '80's, '90's. Schools were banning wearing red and blue because those gangs had hijacked those two colors. Anyone not in those gangs would take care to not wear anything containing those colors.

The op is not wrong for supposing it's possible for someone to be targeted due to wearing "shoe or shirt color".