In addition to the inherent physical nature of the work, another problem is working for small businesses that fly under the radar of OSHA and other regulations. Also, family owned businesses don't have a "normal" career advancement process unless you belong to the family.
When people come to work on my house, I've noticed that everybody who is around my age (58) is hobbling and broken.