Some people are using e-mail like IM and e-mail is built as an asynchronous communication system. In my opinion, that's where the friction happens.
The author of this post brings up a great point about defining expectations on reply times. I think it's healthy to create an auto-reply that states that you only check e-mail x number of times a day.