Even formal mathematical proofs "appeal to authority" when they invoke theorems without re-deriving them every time. It's a shorthand reference to knowledge that time and energy has already been spent discovering. The parallel for this in society, where the present cumulative scope of knowledge is completely impossible for a single person to independently discover on their own, is called expertise. When you are denigrating experts, you are denigrating acquired, shared knowledge.

Pure reason has limits, but one of the neat things about rational argumentation is that it can be applied to itself to discover what alot of its own limits are.