no one complains, and my experience is entirely anecdotal but i've had situations where turning down a drink or the next round was met with, "what's wrong with you?"-esque inquires.
Sometimes, if I am with people who I know longer, I tell them it is because of having an alcohol related death in the family.
I also continue with saying I don't mind that they drink.
I have done this for about 11 years without any issues.