There are no doubt many ways to address this feeling, but it helps to understand what is the underlying fear driving it, and addressing that. Even without knowing what is the root fear, though, maybe it'll help to remind yourself of a few things:
- You're entitled to recreation/recuperation/relaxation.
- You don't need permission to take a break, socialise, etc.
- You don't need to cram "productivity" into every nook and cranny of your life.
- The world will probably get by without your contributions.
- Sometimes something significant comes out of seemingly insignificant or mundane.
It's worth paying attention to that last one. The fact is, we often don't truly know the value of the time we spend until we have the luxury of hindsight. You might blow-off being "productive" one evening to go socialise and end up having a conversation that completely changes your life–or perspective—in some positive meaningful way.
Or, to provide a more common and grounded example: you get stuck on some problem and feel like you want to keep hammering away at it, but for whatever reason end up stepping away a while. When you reluctantly return to it, you figure out a solution in no time at all (at least compared to your earlier efforts).
Give yourself a break. Don't be too hard on yourself. It's okay to be ambitious or to want to keep growing—that's healthy. But life needs balance. Try to temper any self-talk that makes you feel bad about yourself or how choose to spend your time.
EDIT: formatting.