You should really start reading them. Many contain valuable information about the candidate and would help you do you job more effectively. I also work as a hiring manager and would probably have discounted more than one really good hire if I'd not read the cover letter and gained context that didn't fit into their CV.