This brain imaging study:
http://www.psychologytoday.com/articles/199907/the-differenc...
Indicates that the default mode for the introvert brain is to keep the parts of the brain responsible for internal stimulation busy. In extroverts it seem default mode is to pay attention to external stimuli.
While unpleasant social interaction will force everyone to take a break, this would explain why introverts like me find even very pleasant social interaction tiring. It just takes energy to pay attention to external stimuli. Extroverts don't have to spend that extra energy.
That's why one of the tell-tale signs of real introversion is a dislike for music while working. It doesn't matter if it's your favorite music, it's distracting. But for extrovert it would not be.
Born introverts are also a tiny minority of the total population, and this combined with everything else makes me suspect extroverts with social anxiety are in fact most of these "introvert" cases.