That is not a good comparison. It's not ok to pass casual racist comments.
Casual racists comments and jokes fall into the micro-aggression category.
Next time, I hope you build up the courage to tell your aggressors that you are not ok with racist comments.