As a manager, I was constantly dealing with things way outside my control with much more severe consequences: other people's emotions, people quitting, budget cuts, layoffs, political posturing. At one point I was told to layoff an entire team of very talented, very productive Ukrainian engineers because Russia had decided to invade their country and my company didn't want to deal with the _possibility_ that they'd become unproductive.
I know it's entirely dependent on the type of work and the organization, but from a stress-reduction standpoint I'll take staff engineer over people manager every time.