I think a lot of people can sympathize, but as you said, it's kinda taboo to say you wish you didn't have kids! I think you get the same feeling, a loss of freedoms, from any significant evolution in life.

Leaving college: "I wish I could still go out drinking every other day and perform the best in my peer-circle"

Getting to a more serious romantic stage in your relationship: "I wish I could still play video games all weekend and not have to think about another person's feelings"

And in my case, getting a puppy: "I wish I could leave the house for more than a day without having to worry about the living being I'm responsible for"

And then having kids, and then getting old enough that you can't be as active as you want, and then just being an actual grandpa.

I think it's just important to put things in context and look at the sum of the joys in your life. I was pretty stoked in college because I could do whatever I wanted, but I also couldn't afford nice vacations, casual plane trips to see friends, quality (expensive) food, etc.