I would definitely consider myself a nerd and I never got in a fight or physically hurt. I can't think of any friends that did either.
The vast, vast majority of violence that happened at my school (and we had a fair amount) was between unpopular, non-nerdy kids with homelife problems.