Once you have a condition where you need to move left, out of the bike lane, it's a big step. But depending on your route, you may need to avoid pavement issues, trash or other debris in the bike lane, vehicles crossing the lane, or you may need to turn left or pass slower bicycles. And the aditional problem that if there's a marked bike lane, drivers will expect bicycles to remain in it.
Sharrows don't provide subjective safety, they just remind everyone that bicycles are allowed there, and should be expected there. (Of course, when they're marked too far to the right, it's worse than not having any markings)