It's crazy isn't it. Did your higher-ups think he was doing a better job because he was working more on calls all day, where you were off them quick? In my job as soon as you got off a call you got right back on another one with a new customer. So I would have like 2-3x the calls per day of another person. I think that is one of the reasons the other people took so long, they figured whats the point, this person is nice and old I will just stay with them over getting a rude person. Who knows the logic behind it all. I tell you what though, I have some incredible stories that I will never forget. Some insanely funny customers and some hilarious co-workers.
When I first got to my job I had quick call times right away. Several supervisors told me they were going to listen to all my calls because they thought I was just hanging up on people. After they listened to them all, they told me to slow down and take more time. I was like, but I am rewarded for doing a better job and having faster call times. Literally that is how bonuses are given. WTF... None of my customers ever said "slow down please to me." I think they didn't want to pay me more for doing a good job. I just needed to do average so I met their agreement numbers with the internet company.