It certainly doesn't help that many companies never inform you if you don't get a position, so if a candidate doesn't get a response quickly, they are likely to assume they were rejected.
It was hard turning folks down, and it pushed us to have very crisp reasons for saying no, but I think it made us better interviewers and more humane in the process.