Where I fell off was that he consistently skewed guests to the right side of the political spectrum. Did he have Bernie Sanders on and express his support for Bernie? Absolutely. But he more often gives a platform to many people I'd consider far, far right -- and in my summation, it's because the right-leaning side of his audience is louder and demands those subjects. Ultimately, it is not a very balanced show. And that's ok, it's his show.
It's not just Joe. In America, what seems "left" to us is actually fairly moderate, politically. I liked having something to listen to that seemed to truly want to find the middle ground, at first. The show has since gone off the rails, IMO.