I agree about some tropical fruits (mangoes and durians especially), but I've had bananas straight off the tree in Southeast Asia and they really didn't taste any different to me.
Another such exception is avocados: http://en.wikipedia.org/wiki/Avocado#Harvest_and_postharvest (second paragraph under Harvest and Postharvest)