I think a fair difference between music/architecture and sports is the relative permanence of the former. An individual structure will be studied 4000 years from now and tell us how an entire culture thrived. Music from the 1700s is still enjoyed today. Most sports games have no lasting effects beyond a week, and when we actually do study ancient sports, we study ancient art and architecture to learn how it was played; individual games within that sport usually have little relevance unless they involve royalty or politics.
It's certainly an important aspect of our culture, but it has a much smaller role than music and architecture. Everybody knows the Empire State Building and Beethoven, but most people don't know who won last week's football game and by what margin.