I think that’s where the confusion is. A virtue signaler doesn’t necessarily have to be a hypocrite, they can just be someone who is motivated to make overt displays of virtue for personal gain.
It seems to be used to dismiss the entire idea being talked about. For example a liberal saying they want relaxed immigration asylum criteria, and a conservative says they're just virtue signalling, and thereby dismisses the topic. In that way it has become a sort of ad hominem attack. In fact it does a disservice to both sides of the argument because there can be very good arguments for strict asylum thay deserve to be considered but get masked behind such dismissive language.
I don't see a problem using the phrase as long as it isn't used to justify a failure to engage in an actual discourse or debate on the topics at hand.