Your manager is not your enemy. Your manager making a joke is not kicking you down, he's trying to breed camaraderie.
Most of the Evil in This World Is Done by People with Good Intentions - https://quoteinvestigator.com/2019/12/19/intentions/
I would argue that breeding camaraderie requires empathy and understanding with other person as opposed to blindly projecting one’s sense of humor upon another.
This is getting strangely personal. If you are joking, well done on the subtlety!