It's pretty common for people to throw out "citation needed" when they don't want to believe or admit something, but can't plausibly deny it. Sort of the same way people use the term "fallacy" these days.
You are free to see harm in these statements but telling me they are always meant as harmful as you mentioned is your own negative reflection. Not mine or ours, we are free to have our own interpretation as well.