Me neither. OS X Mail has a hidden option that you can enable, so that Mail copies just the email address. Copy this into your Terminal and restart Mail:
defaults write com.apple.mail AddressesIncludeNameOnPasteboard -bool false
defaults write com.apple.mail AddressesIncludeNameOnPasteboard -bool false
Can't blame Apple enough for making this a hidden default one has to use the terminal to enable. WHY ?