I don't see any need to apologise for what's clearly meant as a lighthearted joke. Just don't complain when someone comes up with a clever comeback :)
It's cool.
I found it hilarious, but I'm a weird guy. I just can see where he is coming from where he meant it as a snide joke, not a personal attack.