I find it interesting that an admittedly not-that-smart (calls himself a moron regularly) fighter/comedian who just talks with all sorts of people is sufficiently divisive enough to merit such asterisks.
I think it's because he's not that bright and has one of the most prolific podcasts in terms of listeners that he's divisive. He is mostly uncritical of whatever someone says and isn't that quick on the draw to call people out so he's basically a platform for any mildly famous person to go on the show and spew whatever crackpot theory there is for a few hours unchallenged to his huge base of listeners. He himself is not that controversial but he allows on controversial characters unchallenged.
It is mostly because he has hosted some controversial guests such as Milo Yiannopoulos, Jordan Peterson, and Sargon of Akkad. And by hosting them he is accused of giving the alt-right a platform while also not sufficiently attacking their views.