You might be conflating college athletes with high school and middle school athletes.
By the time you reaches college, you typically:
1. have enough data to see how quickly you're improving;
2. are exposed to a much wider pool of talent than you were previously. Chances are there are many people at your level of athletics that are far better than you; and
3. are exposed to a much more difficult environment than high school and middle school. Combined with 1 and 2, many athletes lose the desire to work that hard for so long.
For athletes that aren't in the upper tier of college athletics, reality has set in even earlier.