Totally agree. Plus, I find that hobbies & activities unrelated to my startup can actually help. I'm a big believer in lateral thinking, and one great way to practice lateral thinking is to shift your mind onto other, unrelated topics.
I was reading this article from the founder of Kayak earlier today:
http://onstartups.com/tabid/3339/bid/12604/Startup-Insights-...
and he mentioned something interesting. Despite being a travel startup, he avoids hiring people from the travel industry. I believe that's because he sees value in having people with different modes of thinking.
And in a way, that's kind of what a hobby does for you. It puts you into a different mode of thinking, which can come back and benefit your startup. :)
Just my $0.02.