You're right but that's not really attacking the theory, rather the supposed proposer (ad hominem essentially, mild as it is). People do tend to be very swayed by appeals to "personal credibility" or lack thereof, but it's not substantively related to the plausibility of the idea itself.