> How do you deal with that?
You teach yourself to say no, to the things you don't want to do.
I considered leaving just that pithy reply, because that's really it. But some of the extra context; It's not a bother to ask someone to hold the door they're already going through because your hands are full. Starting a conversation is about as intrusive as that. The vast majority of people don't mind making some small talk, and ontop of that, the majority can make an excuse if they don't have time. You only assume they can't politely decline, because you can't. Once you learn to say no thanks, politely, but explicitly and directly. You'll actually understand and expect others to return the favor.
That's a much more fair way to interact with people too.