There's nothing new about this. Some people stopped associating with you because they didn't like your opinions, a story as old as humanity itself.
Unless they're actually offended by non-violence, in which case they probably did you a favor.
Most would not say they are offended by non-violence, and most would actually advocate non-violence. The problem comes in when you start pointing out that logical and consistent definitions of violence/aggression preclude a lot of the things people value and think noble. To some, mentioning you are Libertarian inherently means you are saying all of their "noble" and "good" ideas should be scrapped because they rely on aggression. Hence the cliched "you want poor people to starve" criticism of Libertarian-like beliefs.
Violence is also a bit of a misleading word, in that it paints a very physical picture. Threat of violence can be seen as aggression, and is on-par with actual violence because the implication is that if compliance is not achieved, the aggressor will exercise violence. Had to clarify a bit, as I did switch the discussion from violence to aggression, which is what libertarianism and anarcho-capitalism is mainly about.
Their values can simply be different, and it is valid for them to disagree with you based on the conclusions they get from those values.
Aggression is defined differently by different groups after all, so the set of actions that a libertarian objects to based on their understanding of "aggression" would be different to a set of actions that someone else would object to based on a different understanding of "aggression".
Obviously you disagree with their conclusion of dismissing libertarianism and blocking libertarians, and that's fine. What's less fine - in my eyes - is to imply that the disagreement comes from the fact that you are the only one who understands some basic Truth about the world and those that disagree are foolish and cliche for disagreeing with you.
That's the sort of thing that I associate with religion, rather than an honest intellectual discussion of political merits of an ideology.