I think it's more about the fact that the addition is publicised while the removal is silent, combined with EdgeRank hiding most of your fb-friends' activity by default, hence "smoothing" the departure.

PersonYouDontCareAbout (PYDCA) adds you, you accept, he enjoys the add, a few days pass, you remove him. Because PYDCA wasn't really paying attention to you (his feed is full of other people's activity, supposedly more relevant to his interests), he doesn't notice. A few months pass, he forgets everything about you. Even if he remembers, he'll just think he never added you in the end. Even if he remembers and knows he's added you, you simply don't bump into each other so much that he might care.

What is really, really awkward is when he adds you, you do this dance, and then FB keeps reminding him every day that he should add you. FB should have a button "this guy doesn't care" and stop trying to make me look awkward.