The ideal for college admissions should be to get a set of students that have the best possible chance to succeed. Unfortunately, a high school student doesn't have much experience with which to base that determination.
Extracurriculars are supposed to show that the student is well-rounded and driven - attributes that have been shown to be positively correlated with success in college and careers.
How would you want colleges to try to determine if a given student will be successful? Because straight academic scores are not the full picture of a person.