Quite plainly there's no rational reason for someone to suspect that in this particular case of a standard boilerplate HR question ("How do you rate yourself?") that they should expect your secret trapdoor answer ("4 or below") should apply rather than the usual, automatic "7-9 is the only right answer").
I'll take your word that it wasn't intended as a joke. But it was intentionally oblique and slippery. All that you accomplish with a "filter" like this is to flush a lot of otherwise potentially perfectly decent candidates. And if they were to find out after the fact that this was where they "failed" your interview (after taking a whole day off work, driving down there, and going through all the other interview rituals) -- they'd be perfectly justified in applying whatever 4-letter words came to their mind in regard to that company and its vaunted "culture".