That’s the problem with making jokes like that in such a position. It makes you seem like a piece of trash, even if you aren’t.
But, they can assume he is all that as much as you can assume he was joking and isn’t a bad person. You can’t prove your side, it’s merely an interpretation given your own biases. Just like people who call him narcissistic or a sadist are basing that on theirs.