FWIW I know a handful of people of all ranges of success in the tech startup world, managing tens to hundreds or thousands.
Some of them are very driven by the stress of management, and seek it out. Their mentality is that a work situation isn't worthwhile if it isn't pushing you to your limit.
But they're the minority. I feel like you either know that you're this kind of person or you're not.
The rest feel burdened by their situation, and would happily trade for yours 300 days out of the year. They not-irregularly make comments, in private to friends, along the lines of "sometimes I wonder if I'm living life wrong? maybe I should be in the park with my kids instead of on the nth investor call or one-on-one with a direct report".
Like most questions, I think you can answer this with a mixture of experiments and introspection. You say you've manage contractors, for example.
Do you ever feel like you wish you had more responsibility for their lives? The urge to give mentorship is real, do you wish you could do that in a professional setting?
You can probably find ways to scratch those itches that are outside your company, too, for what it's worth.
You're in a good position where you get to "choose your own adventure", so you should experiment with it! Document your feelings, find somebody to talk it through with, iterate towards a happier spot. There's always ways to test your curiosities without sacrificing everything, if you get clever with it. :)