If you had seen lots of resumes and references, then interacted with those people later, you'd distrust resumes and references too.
Why not at least give the appearance of trusting while still making internal judgements and while holding some sort of casual conversation about technical topics with the candidate?