If you play the pointing poker game you tell everyone on the team (of people who might have to work on it) to make an estimate, and if they're unsure then to estimate high. Then you can take the mean, or the highest estimate, or talk through the concerns and come to some consensus (the consensus includes things like "we need a lot more information before we can estimate" or "we need to break this up more").
If an individual is estimating the level of effort for work that could be given to any number of people then I think the estimation system needs improving.
I generally agree with the rest of your comment, though.