I've always thought this is due to the fact that baseball has an extremely high skill ceiling, but great overall physical fitness isn't necessary to be good. There aren't really any around anymore, but look at knuckle ballers like Tim Wakefield, his main pitch he only threw like 60mph and just had perfected this really bizarre skill and was relevant. Also so much of it is mental, which is true in any professional sport, but baseball in particular because you have these intense 1-1 mental battles between the pitcher and the batter, or the pitcher and base runner. If you ever read Tony Gwynn write about what goes through his head when he'd steal home base, it's a mental game of chess he's playing with the pitcher and catcher.