I struggle with this question for myself, certainly. I know I could be working on some kind of revenue-generating side project, or volunteering my time on a political campaign I believe in, or working to feed the homeless, but instead I play video games. Is that.. okay? Is it "right" or "good"? I don't know that I have a perfect answer.
I know this much: without some amount of leisure time, I'd go crazy, nor would I be much use in any of my "productive" capabilities; so clearly, there's an amount of leisure time that is a net good for me. I think the question becomes: what is that amount?