Sweet corn, for example, starts converting sugars to starch soon after it's picked. Many people don't really understand why it's called sweet corn, as they've only ever eaten corn that was picked a few days ago. Asparagus I believe is very similar, with significant flavour changes happening rapidly after harvest.
We've enjoyed dozens of varieties of cherries straight off the trees all with wonderfully distinct flavours, but a day or two after picking, they're all merely 'cherry-flavoured'.
William Alexander wrote, somewhat humorously, about the dollar & effort costs of growing produce at home [1] but in addition to unique / superior flavours, advocates will often assert the intangible benefits (a new problem domain to engage the brain, a tighter connection with your food, the near-meditative aspects of some repetitive gardening activities).
Oh, and for seed acquisition, in AU we've got a few really good options - Green Harvest, Diggers, Edens etc - that maintain many of the heirloom varieties that found there way here, but I thought stateside you'd have way more options? Check the Seed Savers Network[2] if you haven't found them already.