I hated the very idea of someone sitting with me at the computer and coding together, I mean that's stupid right ? And I'm going to be super slow compared to usual, and will have to explain stuff, ...
But even if you're only going at 80% of your usual speed, you're going. 100% speed is useless if you stop after 5 minutes. Consider telling the other person about your focus issue, so they don't let you play around too much.
With time, I've seen plenty of other great sides and bonuses in pair programming, but that one alone has been a god send. And after a while you get used to the rythm and get better at coding alone without stopping too.