I love this talk by Kevin Hale about their all hands support at Wufoo. https://www.youtube.com/watch?v=fAnl8ZGHVXU (alternate. https://www.youtube.com/watch?v=wYIH2qTizUU)
The most interesting thing is that they gave every engineer time to fix the $#@! they heard on support. 30% of their effort was on internal tools.
Disclaimers. I also work at Olark. I also think Kevin Hale is (almost) perfect.