The only time it's a productivity hit to me is when I'm procrastinating anyway. I wouldn't worry about that. Focus on making the conversation as interesting as you can.