In my experience it wasn't really the actual notifications and weird work hours that was the problem. The problem was that I was officially the end of the "it's someone else's problem" chain. It's a funny thing about moral hazards and shit rolling downhill: there's always someone at the bottom. If you're on pager duty, you're at the bottom.
So I liked feeling trusted with an important task and I liked ensuring that other people could sleep. But the pager came to represent every wrong thing with everything in the world. I stared at it in revulsion by the end of things. (Yes, I had an actual pager to stare at.)
That's just my personality, though. Your mileage will vary.