From my experience, the hardest part of being a waiter was dealing with customers.
I worked hard physically as a dishwasher or warehouse worker, but had little mental stress.
I only lasted a few weeks as a waiter. Despite the much higher tips, dealing with bad customers was so stressful I couldn't handle it and it significantly affected my studying.
Whereas I never felt that tired when doing physical labor.