Depends on the colleague, I have done that and similar many times previously and had similar back.
The comment is made, we laugh, he has his salad, I eat my burger, we moved on to something else.
(UK)
No it wouldn't. Try spending some time further up north.
I'm in Scotland :-)
and find that offensive?!
No - I wouldn't find that offensive (I've been called far far worse things in a work context) - but I'd be very worried about potentially causing offense, more for the sake of politeness than the HR impact.