Sure, it does not hurt to wear such clothes. But those arguments aren't mutually exclusive, and by always repeating them, public discussions tend to shift towards the arguments that blame the weaker ones (i.e., cyclists, pedestrians), often neglecting the underlying issues at fault (reckless car drivers or unsafe infrastructure).

For instance, in many cases, when a car has injured a cyclist severely, there could have been infrastructure to prevent this (e.g. separate bike lanes, clear road layouts, …). A helmet would not have prevented the accident.

Similarly, unless someone's crossing a dark street at night without looking for oncoming traffic, why should pedestrians have to be wearing reflective gear all the time, when there could be well-illuminated crosswalks, slower driving speeds, speed bumps etc.?

(I'm not claiming there aren't cyclists ignoring red lights or pedestrians randomly crossing roads in unsafe situations … that's usually the next argument.)