Maybe the driver sensed that you were going to give him a bad rating and decided to get out in front of it and make it into a race-based issue?
Ironically, my girlfriend is a visible minority too, so she was extra-incensed about the racism complaint.