It's not. I've never been fitter than when I was doing physical work. The problem was the pay and the powerlessness at the workplace (e.g. awful hours with no notice, management compromises on safety, the expectation that you'll keep silent when management cuts corners or cheats the clients/customers, etc.)
The reason I stopped doing physical work is because physical workers are disrespected by the world, and your bosses would rather shut down and leave the industry than to pay you a dollar more. After I decided I would never work with my hands again, jobs paid a lot more. The less you actually do, the more holy you get. One day I'll just sit in the lotus position, floating two inches above my prayer mat, giving cryptic pronouncements about what other people should be doing. By then, I'll have billions.