Some people get really weird if you approach their house in a way they don't like. I imagine if that happens a few times to you as a delivery driver, you might be more cautious.
The problem is they've probably gone to the wrong house before and met an owner who got defensive.