I like this. I mean any virtuoso musician, olympic gold medalist, chess grandmaster, pulitzer prize, nobel, and fields medal winners. They should all be penalized for too much studying.
It would make me feel so much better about my own personal mediocrity! GIVE ME MY PARTICIPANT AWARD, DAMMIT!
Also, I don't think most individual exams in courses have enough test-retest validity for an individual score to be a strong signal that either of these problems exists. For a course with a large number of exams (more typical of secondary than higher ed, IMO) you might diagnose a problem by a consistent pattern of very high scores, but an individual score is too weak of a signal.
If you penalize high grades, people will try to avoid them by intentionally dropping answers if they feel too good, and then the utility of your scores for assessing anything, goes down.
An exam is partially meant to measure how well the student learned the information. Why would you want to penalize students doing well?
Imagine being required to go over the same material, year after year, and live in fear of admitting you know it back and forth.
Is this not the public school experience many have?