Right, I don't think we really disagree about anything. "That's an appeal to authority" is not a useful objection, because appealing to authority is often a great idea. "That's not a good authority" is a good objection and often very relevant.

It's sort of like seeing a lot of arguments that rely on false or misleading evidence and deciding that "appeal to evidence" is a fallacy. The choice of authority or evidence is the issue, not the act of appealing.