The thing is, your first example, the black candidate that was passed over for valid reasons wasn't racism. But when you extrapolate to say: "Therefore I won't hire minorities, because they cause extra hassles", you have just jumped straight into the racism pool.

You don't know that any individual is going to correspond to the racial stereotype that you have in your head. It is not fair to treat an individual on the basis of what you believe to be racial characteristics. For starters, your prejudice may be ill-informed, that is, it may not be true, but a result of faulty confirmation-bias on your part. This is extremely difficult to avoid without actually doing a scientific study. And secondly, even if your racial profile is 100% correct, you still can't use that to justify a hiring decision of a candidate, because the candidate is an individual, and may not correspond to the racial profile.