I started sailing in high school and I was fat and out of shape but I was good and I was told I could have been very good. The joy of sailboat racing as the comment above made very clear, it's a brain game. If you're interested you can learn to sail, and race, pretty much anywhere with water, most places have some club or group that is racing Lasers, it's a good place to learn, how fit you need to be depends how far you want to go but it's not the primary factor in being good as far as I can tell (I'm an ok sailor these days, but some of the folks in here are clearly very good).

Counterintuitively, the bigger the boat the less the fitness requirement.

Small boats (dinghies) require crew weight in certain places (“hiking” as far out to windward as possible) and have less mechanical advantage in the boat systems.

Larger boats, the forces scale out of the human range quickly and the crew relies on winches and pulleys to move the sails.