To reply that "hotness is relative" without clarifying which usage of the word you mean or implying that your friend in the anecdote even made that clarification is exactly what the person you are replying to is complaining about.
This limitation of colloquial English where the word "spicy" is overloaded to refer to all three axis of "spicy" at the same time and hot is also (though less frequently) overloaded in the same manner is exactly what we're discussing.
You can have something that's not very "hot" be very "spicy" on one of the other two axis. Try eating a tablespoon of cinnamon or that Chinese mustard that comes with takeout.
European food that is "spicy" on one axis but nearly zero on the other two is rare hence why this overloading exists and isn't really a problem when discussing food from anywhere that doesn't have a rainy season.