Have you tried asking? I asked a few drivers, at the end of a ride and after I’d rated them, what I could do better. Two complained about my taking work calls in the car. I started requesting permission, at the beginning of the ride, to take a phone call. (It’s their car, after all.) Never been refused. Rating jumped from ~4.4 to ~4.9.
I’ll also note that some of the sweetest people I know have terrible ratings for constantly being late, at the wrong pick-up spot or other little reasons. It’s mostly a rating around if you’re observant and treat the driver like a human being more than if you’re a good person.
(Note: I’m a guy. I don’t think gender is as powerful as other comments assume.)