It should be pretty clear if you look at Joes career outside of the podcast. For 30 years he has been in the entertainment industry with stand up comedy, television (News Radio and Fear Factor), to announcing UFC fights.
He was one of the fist big podcasts and livestreamers before others started doing the same. Most of his initial (and repeat) guests are simply friends from these various industries.
He was already a celebrity before he started his podcast.
Unless you consider Joey Diaz to be a “such good guest,” he didn’t
Reread what you responded to. I'm sure it is a large component.
Another question being, since those same guests have been present on many other podcasts (most do rounds for promotions), why does his podcast seem to be preferred?
He let his guests talk without trying to put words in their mouths. He respects his guests and takes a sincere interest in them. In other words, he's the exact opposite of most people in the media.