My experience is anecdotal as well, but it confirms your hunch about collegiate athletes. The have been the best performers, but it's a small sample size in my case. It would be interesting if there is a different correlation between sports (e.g., individual vs. team sports).
The main complaints I've heard about the prestigious institution hires are:
1) They tend to excel well when given a problem that can be solved with a rather templated approach, but tend to struggle more with poorly defined problems
2) They tend to have higher turnover, with the speculation that they jump ship as soon as a perceived higher status opportunity arises. Meaning, they start a lot of projects but don't see them to completion
I don't know if I've had enough experience with the differing groups to draw strong conclusions one way or another.