Sure, it is harder to be taken seriously when even a few people see you as a crackpot.
On the other hand, if you're not telling the truth you'll never be taken seriously by anyone who matters irrespective of how many people do or do not see you as a crackpot.