The most irksome thing about my work during my recent vacation was lugging my laptop everywhere. Apparently I banged it against something and now Dell is busy fixing it for me for $500. Grr. But other than that, it certainly wasn't preventing us from having a nice time.
Also, there are a lot of activities which aren't as critical as you think they are. I was used to doing bug fixes the day they were reported. That's nice, but that's not a law of nature, and you probably don't want to make your customers think it will always happen. If someone reports a bug during vacation and it isn't killing someone or taking the site down just tell them "Thank you. I'll see to it after I get back from vacation."