Of course, it depends on the person. If the person doesn't want lunch, and the boss keeps singling you out to go, that kinda disrespects you. And vice versa.

People are not all the same. Some people are shy, some people are gregarious. Respect their nature, that's it. When someone who has power over you just disregards you as a person, that's both not professional AND abusive if it happens over and over again. It's not a hard concept to get.