I’m a manager now, and have been for years, but leaving product building behind was one of the hardest things I ever did. I did it because I wanted to make decisions more than I wanted to build things, but that’s not for everyone and it’s not as glamorous as it sounds. I could just never help myself from getting involved in every decision, and then it’s frustrating not to have any real weight, and you won’t as “just an expert”, even when you’re right. A lot of people prefer to just build stuff, and in CS and engineering especially, those people aren’t easy to come by or easily replaceable, and thus they are valuable.
My free time is still spent doing silly CS stuff though, I recently discovered edX.org, and I’ve been fooling around with cryptography.
So you’ll certainly have time for that, even when you get older. The only time I didn’t have time was when my kids were 0-6.