I think this has to do with seniority. When I was a lot more experienced than my coworker, doing pair programming was insanely fun. Because it felt like solving a puzzle with a friend, or exploring a video game with my sister. Now I'm occasionally pair programming with my manager -- who's like a 1000x me -- pair programming is mentally blocking. It's because my peer thinks so much faster than I do that it feels like I'm watching an action movie on 10x speed except every 3 minutes someone pauses the movie and asks questions.
EDIT: To make myself clear, this is not a criticism of my boss, it's just that pair programming with someone who is a lot more experienced than you is just naturally intimidating.