Does your employer pay you for hours or output? I'm a salaried employee, I get paid for output, not the time it takes to produce.
I think if anything you're stealing from yourself, robbing yourself of the opportunity to take on more responsibility, grow faster, etc.