Anybody have stories about negative consequences after they decided to leave at a normal time instead of staying late?

I leave at 5, assuming everything's in order, and nobody has ever mentioned anything to me. I also like to get in at 9, even if most other people arrive around 10 -- I love that quiet time in the mornings.

One thing I have noticed is that some people who work late are stuck in a horrible cycle of coming to work late, slacking off, then trying to make up for it by staying in until 7 or later.