I just got back from vacation, and I agree. The whole process of taking pictures with my phone feels so cumbersome and annoying, especially for short lived scenes. It's locked and in my pocket for security and safety because I don't like to walk around holding my phone when I'm trying to enjoy the moment. It feels like I could either have the phone be an appendage and then have fairly easy picture capability, or actually be present in the moment but then it's substandard for taking impromptu pictures.
I wouldn't have as much of a problem with a dedicated device, as I wouldn't necessarily be as worried about breaking or giving someone access to the device that contains details about every aspect of my life.
Alternately, something designed to be an appendage but in an unobtrusive way (to the user, at least) might be just as good or better, as you suggest.