It's the same vibe as "They called <insert genius> crazy for his ideas!"

Yes, the also called lots of actual crazy people crazy. In fact the probability of actually being crazy, given people are calling you crazy, is very high.