This manager wants to make you smaller, so that he can look bigger and exert more power. In my experience, the only way to counter is by also making a strong gesture.
For the dialog you sketched, possible responses could be "Then you can surely do it yourself in 2 minutes". Another, more aggressive, option would be to answer "this joke gets old". You need to find your own tone, obviously, which should not be agitated-aggressive, but make clear that he now has an opponent, and you will not be an easy victim for his power games.
Obviously, this is a risky approach; only do it if you can afford switching jobs. But in my experience it is less risky than it looks. Bullying persons have a good sense of their opponent's strength, and will rather look for new easy targets than going into a serious fight. If you can talk to your team mates and make this a team stance, this will of course strengthen your position further.