The person simply sounds defensive, trying to convince themselves that not having fun was fun. So the reply made a good point.
Imagine: You have a different notion of happiness than most others, so people constantly tell you that you're not happy — even though you are. Naturally, you'll get defensive over time, but then those same people will see that as a sign that you're in fact not happy. Catch 22.
In other words: stop being condescending.