I don't have the same experience at all. The people who I know and have met which enjoy watching sports come at all levels of fitness. I am not that interested myself but quite many of the members of my running club enjoy watching sports and some of them can run 3 hour marathons.

And if you have ever met football ultras most of them are very fit.