I think you're right - according to category theory, only relationships of the objects matter, not the objects themselves.
However, it's just an inspiration and it must be proven in its newly applied field - proof in context of category theory is not valid in the context of inter-personal relations.
Perhaps that serves to illustrate why it is rude. If that doesn't help, then just remember that it is. Good manners don't always need to make sense to you, sometimes they only make sense to other people.