However, commencement speeches do not have a high reputation for academic rigor to start with, so I think inviting an entertainer to speak at one is sort of different from inviting a speaker with some specific experience (either academic, or political/industrial/etc.) to give a lecture.
All that said, keep that Yiannopoulos wanted to read out the names of students without legal immigration status on stage at Berkeley. That's the sort of thing I mean by "antics." I don't think Stephen Colbert does anything of the sort. His partisanship and his satirical character mean that I have trouble saying that anything of value will be lost if he stops being invited to speak at college campuses, but he seems qualitatively different from Yiannopoulos.
I don't think this was ever true, and your whole stance on this hinges on it.