Are you familiar with his research? As much as I dislike affirmative action, I find it difficult to accept that as justification for writing off his work.
I don't know much about his work, and I fully realize it is possible that he might have gotten the grant even without minorities (in contradiction of the statement he made).
In that case, his statement is entirely pointless, since he would have gotten the grant with or without a physics undergrad program.
Besides, even OP states that affirmative action was only "in part" responsible for funding success.