'First class citizen', while not really used outside of programming, was likely meant in contrast to a 'second class citizen.'
"A second-class citizen is a person who is systematically discriminated against within a state or other political jurisdiction, despite their nominal status as a citizen or legal resident there." - Wikipedia
He's saying you probably did not suffer much at all because of systemic discrimination, and pointing out the glaringly obvious -- that you don't realize that majority of the people in this country DO suffer because of systemic discrimination.