The biggest problem, doing that, is that you increase the pressure from bad actors to websites that are trusted by lots of people.
So, for instance, the more weight you give to sites that are quoted by Wikipedia in your search rankings, the more content farms will have incentives to sneak edits that link to their sites.
There are ways to counter that (eg moderation), but in general, defense is more expensive than offense.