The natural (and probably correct) assumption is that physically demanding work causes higher fertility. While "having lots of children leads to physically demanding work" is super unlikely, it's worth considering whether there's some third factor that leads to both high fertility and a preference for physically demanding work.