On a tangent, why is this problem soooo pervasive? I would say the large majority of people I work with fall into the over-confidence bucket.

Perhaps it's that people in tech are smart, and usually right when they find themselves in an argument. Perhaps it's that the hiring process at many companies favors those who display egregious confidence.