It would be very difficult to dispel the notion of prestige, because most output measures are highly influenced by input measures. Prestige inevitably flows into the measures.

E.g. an academic at a prestigious university has a healthy supply of able PhD students, post-docs, a "research environment" that will make applying for grants that bit easier, etc. Their publication and citation numbers will quickly diverge from their identical twin who has is less well resourced. Likewise, the PhD students at a prestigious university are more likely to be attached to well-funded grants, collaborators who have well-tuned paper mills, etc.

Academia is quite a social game. Network effects (which are one part of prestige) are a strong influence.