Interesting - I would guess you usually hold it with your right hand? Is your hearing noticeably asymmetric?
Come to think of it, if I'm doing something else at the same time - such as, of course, writing - I will hold the phone to my left ear with my left hand. Whichever way, in my case, it has little to do with my hearing.