Just one guy's opinion, but to me it sounds like you're making intelligent, healthy, and entirely logical choices about your life and how you choose to spend your limited time.
the hardest part is the tension with the external guilt - but that is the point of the guilt after all