The parent was making a joke, but one way to answer your question might be:
You feel bad because you assume they - like you - feel something when you mistreat them, and you assume this at a very deep level that might not, itself, feel like anything to you.