I don't watch John Oliver's show, but I don't need to... I can pretty confidently say things like:
* John Oliver hasn't told anyone that ivermectin is 99% effective but big pharma doesn't want you to know about it.
* John Oliver hasn't told young people that "I wouldn't get vaccinated"
* John Oliver hasn't blamed a white nationalist rally on the "feds"
* John Oliver didn't have a psychologist on his show to talk about how "there's no such thing as climate"
It has a lot to do with your comment... it's not an appeal to authority when the guy himself says he's a moron and no one should follow his advice. When he says something like that it leaves no choice but to listen to authority because he's clearly stated that he is not authoritative.
We're not talking a single authority here either... hundreds of doctors, including epidemiologists, have complained about Rogan's covid rhetoric... a subject he quite literally has no knowledge about.
You can't waive away all authority as a logical fallacy, often it refers to a false authority or a lack or corroboration. This isn't one of those subjects.