I've been doing regular week-long retreats for 15 years and recently did a 30 day retreat. Personally I just think there's way too much variation amongst my experiences, let alone if you include the gamut of others', to come to any solid conclusions. For example I've met people that meditate more than me that I consider a social liability. Correlation does not imply causation and all that. Are coffee drinkers more productive or are possessors of palettes predisposed to coffee's chemicals/culture more productive?
What is the magic ingredient in meditation that the mainstream seem to be missing out on!? I'm sticking my head out and firmly stating: nothing. All the scientific and anecdotal evidence derives purely from the sober and unromantic intention to pay attention to something -- how you pay attention, the sincerity with which you pay attention, the subconscious emotions and intellectual knots illuminated from such attention, these are the factors that determine the various reports we find concerning the 'magic' of meditation.